2024-02-10T00:50:02.057 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T00:50:02.057 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --owner scheduled_yuriw@teuthology --name yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696 -- /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/orig.config.yaml 2024-02-10T00:50:02.104 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T00:50:02.197 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696 branch: main description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: yweinste@redhat.com first_in_suite: false job_id: '7554696' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5679 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlSAKYrhH4t6lqydJwPc7dakwrUcEIWsSCqIlLXUX8KcGMzn9wmRKv3395yFBqkRi00GYbssmuqVssZshiM65E= smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3fOJ0AL93zBAlc9KUWLcYTG+/ZN22p+OeOUdu5BJmCIqUtDwSyLdMZQynzTw6sRiugO8+IUpLWGgYqOxmkkzs= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:39:00 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T00:50:02.197 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T00:50:02.198 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T00:50:02.198 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T00:50:02.200 INFO:teuthology.task.internal:Checking packages... 2024-02-10T00:50:02.224 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T00:50:02.224 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T00:50:02.224 INFO:teuthology.packaging:ref: None 2024-02-10T00:50:02.224 INFO:teuthology.packaging:tag: None 2024-02-10T00:50:02.224 INFO:teuthology.packaging:branch: main 2024-02-10T00:50:02.225 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T00:50:02.225 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T00:50:02.582 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T00:50:02.584 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T00:50:02.590 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T00:50:02.590 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T00:50:02.620 INFO:teuthology.task.internal:Saving configuration 2024-02-10T00:50:02.637 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T00:50:02.655 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T00:50:02.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 00:45:09.209043', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlSAKYrhH4t6lqydJwPc7dakwrUcEIWsSCqIlLXUX8KcGMzn9wmRKv3395yFBqkRi00GYbssmuqVssZshiM65E='} 2024-02-10T00:50:02.701 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 00:45:09.207217', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3fOJ0AL93zBAlc9KUWLcYTG+/ZN22p+OeOUdu5BJmCIqUtDwSyLdMZQynzTw6sRiugO8+IUpLWGgYqOxmkkzs='} 2024-02-10T00:50:02.702 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T00:50:02.712 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-02-10T00:50:02.713 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-02-10T00:50:02.713 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T00:50:02.817 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7848131430>, signals=[15]) 2024-02-10T00:50:02.817 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T00:50:02.828 INFO:teuthology.task.internal:Opening connections... 2024-02-10T00:50:02.828 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T00:50:02.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:50:02.936 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T00:50:02.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:50:03.045 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T00:50:03.057 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-02-10T00:50:03.075 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-02-10T00:50:03.075 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="9" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="9" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el9" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-02-10T00:50:03.133 INFO:teuthology.orchestra.run.smithi008.stdout:LOGO="fedora-logo-icon" 2024-02-10T00:50:03.134 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T00:50:03.134 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-02-10T00:50:03.134 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T00:50:03.134 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T00:50:03.134 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T00:50:03.134 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-02-10T00:50:03.161 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-02-10T00:50:03.179 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-02-10T00:50:03.179 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-02-10T00:50:03.236 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-02-10T00:50:03.236 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-02-10T00:50:03.236 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-02-10T00:50:03.236 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T00:50:03.237 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T00:50:03.237 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-02-10T00:50:03.260 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T00:50:03.274 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T00:50:03.335 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T00:50:03.335 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2024-02-10T00:50:03.338 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-02-10T00:50:03.354 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T00:50:03.373 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T00:50:03.373 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2024-02-10T00:50:03.395 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-02-10T00:50:03.429 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T00:50:03.486 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T00:50:03.506 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T00:50:03.506 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T00:50:03.506 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T00:50:03.507 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T00:50:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T00:50:03.524 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-02-10T00:50:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T00:50:03.525 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-02-10T00:50:03.582 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2024-02-10T00:50:03.583 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-361.el9.x86_64 2024-02-10T00:50:03.583 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-361.el9.x86_64 2024-02-10T00:50:03.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-02-10T00:50:03.583 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.14.0-361.el9.x86_64 2024-02-10T00:50:03.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum install -y kernel 2024-02-10T00:50:05.680 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T00:50:05.718 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-10T00:50:07.151 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-02-10T00:50:08.958 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-02-10T00:50:12.759 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-02-10T00:50:13.206 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - BaseOS 322 kB/s | 2.0 MB 00:06 2024-02-10T00:50:14.637 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 7.4 MB/s | 8.0 MB 00:01 2024-02-10T00:50:17.212 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 5.9 MB 00:00 2024-02-10T00:50:18.987 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2024-02-10T00:50:20.513 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-02-10T00:50:25.604 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-02-10T00:50:27.975 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T00:50:27.975 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T00:50:28.064 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:50:28.075 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-02-10T00:50:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:50:28.077 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 99 M 2024-02-10T00:50:28.077 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 122 M 2024-02-10T00:50:28.077 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:50:28.432 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-10T00:50:28.956 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-10T00:50:29.714 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 26 MB/s | 33 MB 00:01 2024-02-10T00:50:30.289 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-10T00:50:30.293 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:50:30.293 INFO:teuthology.orchestra.run.smithi060.stdout:Total 45 MB/s | 99 MB 00:02 2024-02-10T00:50:30.293 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:50:30.776 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:50:30.776 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:50:31.809 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:50:31.809 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:50:32.678 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-02-10T00:50:34.330 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:50:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T00:50:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T00:50:36.662 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 16 MB/s | 5.9 MB 00:00 2024-02-10T00:50:38.021 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 15 kB 00:00 2024-02-10T00:50:38.029 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T00:50:38.212 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T00:50:40.059 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-10T00:50:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T00:50:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:50:45.248 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-10T00:50:47.639 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T00:50:47.639 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T00:50:47.736 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:50:47.743 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:50:47.743 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout:Install 4 Packages 2024-02-10T00:50:47.744 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:50:47.745 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 99 M 2024-02-10T00:50:47.745 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 122 M 2024-02-10T00:50:47.745 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:50:48.108 INFO:teuthology.orchestra.run.smithi008.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-10T00:50:48.540 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:50:48.667 INFO:teuthology.orchestra.run.smithi008.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-10T00:50:50.009 INFO:teuthology.orchestra.run.smithi008.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 18 MB/s | 33 MB 00:01 2024-02-10T00:50:50.484 INFO:teuthology.orchestra.run.smithi008.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-02-10T00:50:50.487 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:50:50.487 INFO:teuthology.orchestra.run.smithi008.stdout:Total 36 MB/s | 99 MB 00:02 2024-02-10T00:50:50.488 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:50:50.925 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:50:50.925 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:50:51.984 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:50:51.984 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:50:54.516 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:50:55.189 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T00:50:55.318 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T00:50:58.231 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T00:50:58.423 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T00:51:03.482 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T00:51:03.587 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:08.674 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:19.505 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:19.505 INFO:teuthology.orchestra.run.smithi060.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T00:51:19.505 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:51:19.577 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:19.979 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:19.979 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T00:51:19.980 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T00:51:19.980 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T00:51:20.400 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:51:20.401 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:51:20.927 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T00:51:21.056 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T00:51:21.057 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T00:51:21.057 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T00:51:21.057 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T00:51:21.057 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T00:51:21.057 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-02-10T00:51:21.057 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T00:51:21.057 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T00:51:21.170 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T00:51:21.171 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T00:51:21.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-02-10T00:51:21.219 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T00:51:21.220 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:51:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T00:51:21.222 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T00:51:21.222 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-419.el9.x86_64 2024-02-10T00:51:21.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T00:51:21.865 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-02-10T00:51:23.278 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T00:51:23.326 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-02-10T00:51:23.328 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-02-10T00:51:23.383 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.IAjiuZZKFG 2024-02-10T00:51:23.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IAjiuZZKFG 2024-02-10T00:51:23.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.IAjiuZZKFG 2024-02-10T00:51:23.575 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.IAjiuZZKFG is 6KB 2024-02-10T00:51:23.625 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.IAjiuZZKFG 2024-02-10T00:51:23.640 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T00:51:23.716 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T00:51:23.716 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T00:51:23.716 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T00:51:23.716 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T00:51:23.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T00:51:23.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-02-10T00:51:39.419 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:39.419 INFO:teuthology.orchestra.run.smithi008.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T00:51:39.419 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:51:39.491 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:39.913 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:39.913 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T00:51:39.914 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T00:51:39.914 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:51:40.329 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:51:40.850 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T00:51:40.990 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T00:51:40.990 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T00:51:40.990 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T00:51:40.990 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T00:51:40.990 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T00:51:40.991 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-02-10T00:51:40.991 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T00:51:40.991 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T00:51:41.102 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T00:51:41.103 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T00:51:41.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo rpm -qi grub2-tools 2024-02-10T00:51:41.140 INFO:teuthology.orchestra.run.smithi008.stdout:Name : grub2-tools 2024-02-10T00:51:41.140 INFO:teuthology.orchestra.run.smithi008.stdout:Epoch : 1 2024-02-10T00:51:41.140 INFO:teuthology.orchestra.run.smithi008.stdout:Version : 2.06 2024-02-10T00:51:41.140 INFO:teuthology.orchestra.run.smithi008.stdout:Release : 68.el9 2024-02-10T00:51:41.140 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: x86_64 2024-02-10T00:51:41.140 INFO:teuthology.orchestra.run.smithi008.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Group : Unspecified 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Size : 8264254 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:License : GPLv3+ 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Packager : builder@centos.org 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Vendor : CentOS 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Summary : Support tools for GRUB. 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout:Description : 2024-02-10T00:51:41.141 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:51:41.142 INFO:teuthology.orchestra.run.smithi008.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T00:51:41.142 INFO:teuthology.orchestra.run.smithi008.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T00:51:41.142 INFO:teuthology.orchestra.run.smithi008.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T00:51:41.142 INFO:teuthology.orchestra.run.smithi008.stdout:hardware devices. 2024-02-10T00:51:41.142 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:51:41.142 INFO:teuthology.orchestra.run.smithi008.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T00:51:41.143 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T00:51:41.143 INFO:teuthology.task.kernel:Updating grub on smithi008 to boot 5.14.0-419.el9.x86_64 2024-02-10T00:51:41.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T00:51:41.725 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2024-02-10T00:51:43.182 INFO:teuthology.orchestra.run.smithi008.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T00:51:43.218 INFO:teuthology.orchestra.run.smithi008.stderr:done 2024-02-10T00:51:43.220 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-02-10T00:51:43.235 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.lmETHCSaJM 2024-02-10T00:51:43.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lmETHCSaJM 2024-02-10T00:51:43.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.lmETHCSaJM 2024-02-10T00:51:43.454 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.lmETHCSaJM is 6KB 2024-02-10T00:51:43.474 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.lmETHCSaJM 2024-02-10T00:51:43.489 DEBUG:teuthology.orchestra.run.smithi008:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T00:51:43.558 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T00:51:43.558 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T00:51:43.558 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T00:51:43.558 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T00:51:43.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T00:51:43.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2024-02-10T00:51:53.880 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-02-10T00:51:53.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-02-10T00:51:53.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:52:12.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-02-10T00:52:13.724 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-02-10T00:52:13.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T00:52:13.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:52:21.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-02-10T00:52:21.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:52:32.191 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T00:52:41.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T00:52:41.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:52:44.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T00:52:56.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T00:52:56.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:53:21.257 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T00:53:28.076 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-02-10T00:53:28.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T00:53:28.528 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T00:53:28.528 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-02-10T00:53:28.542 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-419.el9.x86_64 2024-02-10T00:53:28.542 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T00:53:28.542 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T00:53:28.542 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T00:53:29.543 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-02-10T00:53:29.544 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T00:53:29.644 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2024-02-10T00:53:29.680 DEBUG:teuthology.parallel:result is None 2024-02-10T00:53:33.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-02-10T00:53:33.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:53:33.555 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-02-10T00:53:33.957 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-02-10T00:53:33.957 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T00:53:33.958 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-02-10T00:53:33.973 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-419.el9.x86_64 2024-02-10T00:53:33.973 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T00:53:33.973 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T00:53:33.973 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T00:53:34.974 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-02-10T00:53:34.974 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T00:53:35.055 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-02-10T00:53:35.086 DEBUG:teuthology.parallel:result is None 2024-02-10T00:53:35.086 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T00:53:35.094 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T00:53:35.095 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T00:53:35.098 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T00:53:35.114 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T00:53:35.123 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T00:53:35.157 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T00:53:35.157 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T00:53:35.160 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T00:53:35.197 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T00:53:35.234 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T00:53:35.234 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T00:53:35.236 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T00:53:35.268 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T00:53:35.271 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T00:53:35.282 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T00:53:35.283 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T00:53:35.285 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T00:53:35.291 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T00:53:35.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T00:53:35.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T00:53:35.356 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T00:53:35.366 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T00:53:35.366 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T00:53:35.397 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T00:53:35.413 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T00:53:35.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T00:53:35.542 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T00:53:35.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T00:53:35.668 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T00:53:35.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T00:53:35.735 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T00:53:35.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T00:53:35.834 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T00:53:35.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T00:53:35.963 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T00:53:35.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T00:53:36.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2024-02-10T00:53:36.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-02-10T00:53:36.085 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T00:53:36.114 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T00:53:36.621 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T00:53:36.629 INFO:teuthology.task.internal:Starting timer... 2024-02-10T00:53:36.629 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T00:53:36.903 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T00:53:37.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo service auditd rotate 2024-02-10T00:53:37.072 INFO:teuthology.orchestra.run.smithi008.stdout:Rotating logs: 2024-02-10T00:53:37.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-02-10T00:53:37.124 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-02-10T00:53:37.127 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T00:53:37.127 DEBUG:teuthology.orchestra.run.smithi008:> /usr/sbin/getenforce 2024-02-10T00:53:37.157 INFO:teuthology.orchestra.run.smithi008.stdout:Permissive 2024-02-10T00:53:37.158 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-02-10T00:53:37.177 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-02-10T00:53:37.177 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi008.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi060.front.sepia.ceph.com': 'permissive'} 2024-02-10T00:53:37.177 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T00:53:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T00:53:37.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T00:53:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T00:53:37.253 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T00:53:37.253 DEBUG:teuthology.orchestra.run.smithi008:> sudo /usr/sbin/setenforce permissive 2024-02-10T00:53:37.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-02-10T00:53:37.329 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T00:53:37.504 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-10T00:53:37.509 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-10T00:53:37.652 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T00:53:37.667 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T00:53:37.669 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T00:57:53.972 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2024-02-10T00:57:53.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T00:57:53.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:57:54.068 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-02-10T00:57:54.153 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T00:57:54.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-02-10T00:57:54.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T00:57:54.245 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-02-10T00:57:54.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-02-10T00:57:54.332 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T00:57:54.343 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T00:57:54.343 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T00:57:54.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T00:57:54.346 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T00:57:54.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T00:57:54.379 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T00:57:54.397 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T00:57:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T00:57:54.422 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T00:57:54.428 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: ntpd: command not found 2024-02-10T00:57:54.443 INFO:teuthology.orchestra.run.smithi008.stdout:506 Cannot talk to daemon 2024-02-10T00:57:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-02-10T00:57:54.460 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T00:57:54.466 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-02-10T00:57:54.477 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T00:57:54.482 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T00:57:54.499 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T00:57:54.536 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-02-10T00:57:54.539 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T00:57:54.539 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-02-10T00:57:54.539 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.540 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.540 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.540 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.549 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-02-10T00:57:54.552 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T00:57:54.552 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-02-10T00:57:54.553 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.553 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.553 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.553 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T00:57:54.553 INFO:teuthology.run_tasks:Running task install... 2024-02-10T00:57:54.561 DEBUG:teuthology.task.install:project ceph 2024-02-10T00:57:54.561 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-10T00:57:54.561 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T00:57:54.561 INFO:teuthology.task.install:Using flavor: default 2024-02-10T00:57:54.567 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T00:57:54.567 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-02-10T00:57:54.568 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T00:57:54.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T00:57:54.570 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T00:57:54.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T00:57:54.725 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-10T00:57:54.725 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-10T00:57:54.742 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-10T00:57:54.742 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-10T00:57:54.844 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T00:57:54.845 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T00:57:54.845 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T00:57:54.870 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T00:57:54.870 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T00:57:54.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T00:57:54.873 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-02-10T00:57:54.873 DEBUG:teuthology.orchestra.run.smithi008:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T00:57:54.898 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-02-10T00:57:54.898 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T00:57:54.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T00:57:54.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T00:57:55.024 DEBUG:teuthology.orchestra.run.smithi008:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T00:57:55.044 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T00:57:55.091 INFO:teuthology.orchestra.run.smithi008.stdout:check_obsoletes = 1 2024-02-10T00:57:55.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum clean all 2024-02-10T00:57:55.110 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-02-10T00:57:55.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-02-10T00:57:55.354 INFO:teuthology.orchestra.run.smithi008.stdout:74 files removed 2024-02-10T00:57:55.369 INFO:teuthology.orchestra.run.smithi060.stdout:74 files removed 2024-02-10T00:57:55.381 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-radosgw 2024-02-10T00:57:55.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2024-02-10T00:57:56.063 INFO:teuthology.orchestra.run.smithi008.stdout:ceph packages for x86_64 228 kB/s | 82 kB 00:00 2024-02-10T00:57:56.098 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 215 kB/s | 82 kB 00:00 2024-02-10T00:57:56.739 INFO:teuthology.orchestra.run.smithi008.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-02-10T00:57:56.790 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-02-10T00:57:57.373 INFO:teuthology.orchestra.run.smithi008.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-02-10T00:57:57.399 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2024-02-10T00:57:57.809 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-02-10T00:57:57.918 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T00:57:59.318 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-02-10T00:57:59.447 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-02-10T00:58:00.801 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-02-10T00:58:16.951 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:14 2024-02-10T00:58:21.177 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 10 MB/s | 5.9 MB 00:00 2024-02-10T00:58:22.603 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 15 kB 00:00 2024-02-10T00:58:25.207 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 20 MB 00:02 2024-02-10T00:58:26.522 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 305 kB/s | 8.0 MB 00:26 2024-02-10T00:58:30.204 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-10T00:58:31.882 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-02-10T00:58:32.985 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:58:32.997 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:58:32.997 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:58:32.997 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:58:32.997 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:58:32.997 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-10T00:58:32.998 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-10T00:58:32.999 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:58:33.000 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:58:33.001 INFO:teuthology.orchestra.run.smithi060.stdout:Install 30 Packages 2024-02-10T00:58:33.001 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2024-02-10T00:58:33.001 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:33.001 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 58 M 2024-02-10T00:58:33.001 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:58:34.289 INFO:teuthology.orchestra.run.smithi060.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 4.4 MB/s | 5.3 MB 00:01 2024-02-10T00:58:34.397 INFO:teuthology.orchestra.run.smithi060.stdout:(2/32): ceph-radosgw-19.0.0-1361.g290212a1.el9. 7.4 MB/s | 9.7 MB 00:01 2024-02-10T00:58:34.423 INFO:teuthology.orchestra.run.smithi060.stdout:(3/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 188 kB/s | 25 kB 00:00 2024-02-10T00:58:34.481 INFO:teuthology.orchestra.run.smithi060.stdout:(4/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 8.7 MB/s | 743 kB 00:00 2024-02-10T00:58:34.565 INFO:teuthology.orchestra.run.smithi060.stdout:(5/32): libradosstriper1-19.0.0-1361.g290212a1. 3.4 MB/s | 499 kB 00:00 2024-02-10T00:58:34.657 INFO:teuthology.orchestra.run.smithi060.stdout:(6/32): python3-ceph-argparse-19.0.0-1361.g2902 492 kB/s | 45 kB 00:00 2024-02-10T00:58:34.757 INFO:teuthology.orchestra.run.smithi060.stdout:(7/32): python3-ceph-common-19.0.0-1361.g290212 1.3 MB/s | 131 kB 00:00 2024-02-10T00:58:34.849 INFO:teuthology.orchestra.run.smithi060.stdout:(8/32): python3-cephfs-19.0.0-1361.g290212a1.el 1.7 MB/s | 161 kB 00:00 2024-02-10T00:58:35.049 INFO:teuthology.orchestra.run.smithi060.stdout:(9/32): ceph-common-19.0.0-1361.g290212a1.el9.x 10 MB/s | 20 MB 00:01 2024-02-10T00:58:35.074 INFO:teuthology.orchestra.run.smithi060.stdout:(10/32): python3-rados-19.0.0-1361.g290212a1.el 1.4 MB/s | 319 kB 00:00 2024-02-10T00:58:35.100 INFO:teuthology.orchestra.run.smithi060.stdout:(11/32): python3-rbd-19.0.0-1361.g290212a1.el9. 5.8 MB/s | 298 kB 00:00 2024-02-10T00:58:35.125 INFO:teuthology.orchestra.run.smithi060.stdout:(12/32): python3-rgw-19.0.0-1361.g290212a1.el9. 1.9 MB/s | 99 kB 00:00 2024-02-10T00:58:35.217 INFO:teuthology.orchestra.run.smithi060.stdout:(13/32): librgw2-19.0.0-1361.g290212a1.el9.x86_ 6.9 MB/s | 5.1 MB 00:00 2024-02-10T00:58:35.284 INFO:teuthology.orchestra.run.smithi060.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-02-10T00:58:35.309 INFO:teuthology.orchestra.run.smithi060.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 44 kB 00:00 2024-02-10T00:58:35.343 INFO:teuthology.orchestra.run.smithi060.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 829 kB/s | 159 kB 00:00 2024-02-10T00:58:35.368 INFO:teuthology.orchestra.run.smithi060.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-02-10T00:58:35.393 INFO:teuthology.orchestra.run.smithi060.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-02-10T00:58:35.418 INFO:teuthology.orchestra.run.smithi060.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 552 kB/s | 41 kB 00:00 2024-02-10T00:58:35.444 INFO:teuthology.orchestra.run.smithi060.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-02-10T00:58:35.469 INFO:teuthology.orchestra.run.smithi060.stdout:(21/32): qatlib-service-23.11.0-1.el9.x86_64.rp 448 kB/s | 34 kB 00:00 2024-02-10T00:58:35.494 INFO:teuthology.orchestra.run.smithi060.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 575 kB/s | 43 kB 00:00 2024-02-10T00:58:35.720 INFO:teuthology.orchestra.run.smithi060.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-02-10T00:58:35.787 INFO:teuthology.orchestra.run.smithi060.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 974 kB/s | 309 kB 00:00 2024-02-10T00:58:35.812 INFO:teuthology.orchestra.run.smithi060.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 528 kB/s | 49 kB 00:00 2024-02-10T00:58:35.845 INFO:teuthology.orchestra.run.smithi060.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-02-10T00:58:35.904 INFO:teuthology.orchestra.run.smithi060.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-02-10T00:58:35.979 INFO:teuthology.orchestra.run.smithi060.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-02-10T00:58:36.088 INFO:teuthology.orchestra.run.smithi060.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-02-10T00:58:36.197 INFO:teuthology.orchestra.run.smithi060.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-02-10T00:58:36.438 INFO:teuthology.orchestra.run.smithi060.stdout:(31/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-02-10T00:58:36.497 INFO:teuthology.orchestra.run.smithi060.stdout:(32/32): librados2-19.0.0-1361.g290212a1.el9.x8 6.5 MB/s | 3.4 MB 00:00 2024-02-10T00:58:36.504 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:58:36.504 INFO:teuthology.orchestra.run.smithi060.stdout:Total 17 MB/s | 58 MB 00:03 2024-02-10T00:58:36.504 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:58:36.543 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:58:36.543 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:58:37.028 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:58:37.029 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:58:38.695 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:58:38.796 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-10T00:58:38.810 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-10T00:58:38.836 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-10T00:58:38.858 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T00:58:38.875 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T00:58:39.050 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T00:58:39.107 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T00:58:39.454 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-10T00:58:39.512 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T00:58:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T00:58:39.695 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-10T00:58:39.855 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T00:58:39.911 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T00:58:39.947 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-10T00:58:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-10T00:58:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T00:58:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T00:58:40.169 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T00:58:40.197 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-10T00:58:40.310 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-10T00:58:40.347 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-10T00:58:40.395 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-10T00:58:40.422 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-10T00:58:40.451 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-10T00:58:40.484 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-10T00:58:40.815 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-10T00:58:40.847 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T00:58:40.945 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T00:58:40.977 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-10T00:58:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-10T00:58:41.496 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-10T00:58:41.524 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T00:58:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T00:58:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-10T00:58:41.693 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-10T00:58:42.590 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 859 kB/s | 5.9 MB 00:07 2024-02-10T00:58:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T00:58:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T00:58:44.174 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-02-10T00:58:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T00:58:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T00:58:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T00:58:44.660 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T00:58:44.660 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T00:58:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T00:58:45.784 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-02-10T00:58:50.751 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-02-10T00:58:53.524 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout:Upgrading: 2024-02-10T00:58:53.532 INFO:teuthology.orchestra.run.smithi008.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: boost-program-options 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T00:58:53.533 INFO:teuthology.orchestra.run.smithi008.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-10T00:58:53.534 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout:Installing weak dependencies: 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout:Install 30 Packages 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout:Upgrade 2 Packages 2024-02-10T00:58:53.535 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:58:53.536 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 58 M 2024-02-10T00:58:53.536 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:58:54.444 INFO:teuthology.orchestra.run.smithi008.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 7.4 MB/s | 5.3 MB 00:00 2024-02-10T00:58:54.511 INFO:teuthology.orchestra.run.smithi008.stdout:(2/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 377 kB/s | 25 kB 00:00 2024-02-10T00:58:54.611 INFO:teuthology.orchestra.run.smithi008.stdout:(3/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 7.3 MB/s | 743 kB 00:00 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-02-10T00:58:54.661 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:54.703 INFO:teuthology.orchestra.run.smithi008.stdout:(4/32): libradosstriper1-19.0.0-1361.g290212a1. 5.3 MB/s | 499 kB 00:00 2024-02-10T00:58:55.178 INFO:teuthology.orchestra.run.smithi008.stdout:(5/32): librgw2-19.0.0-1361.g290212a1.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-02-10T00:58:55.287 INFO:teuthology.orchestra.run.smithi008.stdout:(6/32): ceph-radosgw-19.0.0-1361.g290212a1.el9. 6.2 MB/s | 9.7 MB 00:01 2024-02-10T00:58:55.312 INFO:teuthology.orchestra.run.smithi008.stdout:(7/32): python3-ceph-argparse-19.0.0-1361.g2902 338 kB/s | 45 kB 00:00 2024-02-10T00:58:55.346 INFO:teuthology.orchestra.run.smithi008.stdout:(8/32): python3-ceph-common-19.0.0-1361.g290212 2.2 MB/s | 131 kB 00:00 2024-02-10T00:58:55.380 INFO:teuthology.orchestra.run.smithi008.stdout:(9/32): python3-cephfs-19.0.0-1361.g290212a1.el 2.4 MB/s | 161 kB 00:00 2024-02-10T00:58:55.421 INFO:teuthology.orchestra.run.smithi008.stdout:(10/32): python3-rados-19.0.0-1361.g290212a1.el 4.1 MB/s | 319 kB 00:00 2024-02-10T00:58:55.447 INFO:teuthology.orchestra.run.smithi008.stdout:(11/32): python3-rbd-19.0.0-1361.g290212a1.el9. 4.4 MB/s | 298 kB 00:00 2024-02-10T00:58:55.480 INFO:teuthology.orchestra.run.smithi008.stdout:(12/32): python3-rgw-19.0.0-1361.g290212a1.el9. 1.7 MB/s | 99 kB 00:00 2024-02-10T00:58:55.606 INFO:teuthology.orchestra.run.smithi008.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 666 kB/s | 106 kB 00:00 2024-02-10T00:58:55.609 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T00:58:55.622 INFO:teuthology.orchestra.run.smithi008.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-02-10T00:58:55.634 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T00:58:55.634 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:58:55.634 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T00:58:55.634 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T00:58:55.634 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T00:58:55.634 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:55.636 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T00:58:55.648 INFO:teuthology.orchestra.run.smithi008.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-10T00:58:55.706 INFO:teuthology.orchestra.run.smithi008.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-02-10T00:58:55.721 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T00:58:55.721 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T00:58:55.732 INFO:teuthology.orchestra.run.smithi008.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-02-10T00:58:55.757 INFO:teuthology.orchestra.run.smithi008.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2024-02-10T00:58:55.782 INFO:teuthology.orchestra.run.smithi008.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-02-10T00:58:55.807 INFO:teuthology.orchestra.run.smithi008.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 676 kB/s | 34 kB 00:00 2024-02-10T00:58:55.833 INFO:teuthology.orchestra.run.smithi008.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 865 kB/s | 43 kB 00:00 2024-02-10T00:58:56.050 INFO:teuthology.orchestra.run.smithi008.stdout:(22/32): ceph-common-19.0.0-1361.g290212a1.el9. 8.5 MB/s | 20 MB 00:02 2024-02-10T00:58:56.200 INFO:teuthology.orchestra.run.smithi008.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-02-10T00:58:56.225 INFO:teuthology.orchestra.run.smithi008.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 739 kB/s | 309 kB 00:00 2024-02-10T00:58:56.251 INFO:teuthology.orchestra.run.smithi008.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 976 kB/s | 49 kB 00:00 2024-02-10T00:58:56.276 INFO:teuthology.orchestra.run.smithi008.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-02-10T00:58:56.318 INFO:teuthology.orchestra.run.smithi008.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-02-10T00:58:56.343 INFO:teuthology.orchestra.run.smithi008.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.9 MB/s | 192 kB 00:00 2024-02-10T00:58:56.418 INFO:teuthology.orchestra.run.smithi008.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-02-10T00:58:56.461 INFO:teuthology.orchestra.run.smithi008.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-02-10T00:58:56.678 INFO:teuthology.orchestra.run.smithi008.stdout:(31/32): librados2-19.0.0-1361.g290212a1.el9.x8 10 MB/s | 3.4 MB 00:00 2024-02-10T00:58:56.769 INFO:teuthology.orchestra.run.smithi008.stdout:(32/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-02-10T00:58:56.775 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:58:56.775 INFO:teuthology.orchestra.run.smithi008.stdout:Total 18 MB/s | 58 MB 00:03 2024-02-10T00:58:56.776 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:58:56.811 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:58:56.811 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:58:56.955 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-10T00:58:56.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-10T00:58:56.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-10T00:58:56.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-10T00:58:56.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-10T00:58:56.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-10T00:58:56.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T00:58:56.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-10T00:58:56.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-10T00:58:57.302 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:58:57.302 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:58:58.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2024-02-10T00:58:58.844 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:58:58.914 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-10T00:58:58.949 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-10T00:58:58.984 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-10T00:58:58.985 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:28 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:58:59.014 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T00:58:59.030 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T00:58:59.154 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:58:59.155 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:58:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:58:59.155 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:58:59.155 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T00:58:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:59.157 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:58:59.157 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:58:59.157 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-02-10T00:58:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:58:59.158 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 48 M 2024-02-10T00:58:59.158 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 204 M 2024-02-10T00:58:59.158 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:58:59.222 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T00:58:59.279 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T00:58:59.379 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-02-10T00:58:59.429 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-02-10T00:58:59.471 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-02-10T00:58:59.513 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-10T00:58:59.538 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 429 kB/s | 163 kB 00:00 2024-02-10T00:58:59.671 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-10T00:58:59.701 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T00:58:59.776 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T00:58:59.866 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-10T00:58:59.895 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T00:58:59.940 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T00:58:59.969 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-10T00:59:00.005 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-10T00:59:00.074 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T00:59:00.097 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T00:59:00.143 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T00:59:00.170 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-10T00:59:00.282 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-10T00:59:00.313 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-10T00:59:00.355 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-10T00:59:00.387 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-10T00:59:00.416 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-10T00:59:00.448 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-10T00:59:00.777 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-10T00:59:00.811 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T00:59:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T00:59:00.949 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-10T00:59:00.996 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-10T00:59:01.477 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-10T00:59:01.506 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T00:59:01.568 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T00:59:01.630 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 19 MB/s | 47 MB 00:02 2024-02-10T00:59:01.633 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:01.633 INFO:teuthology.orchestra.run.smithi060.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-10T00:59:01.634 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:01.638 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-10T00:59:01.656 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:01.656 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:01.666 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-10T00:59:02.063 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:02.063 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:02.666 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:02.744 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T00:59:02.779 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T00:59:02.824 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T00:59:02.864 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T00:59:02.891 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T00:59:03.944 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T00:59:03.996 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T00:59:04.591 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T00:59:04.633 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T00:59:04.724 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T00:59:04.725 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T00:59:04.725 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T00:59:04.725 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:04.768 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T00:59:07.617 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T00:59:07.693 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T00:59:10.074 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T00:59:10.074 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-10T00:59:10.075 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-10T00:59:10.075 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T00:59:10.075 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T00:59:10.075 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-16.el9.x86_64 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:10.453 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T00:59:10.454 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T00:59:10.454 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T00:59:10.454 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:10.454 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:10.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2024-02-10T00:59:11.389 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:41 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:11.560 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:11.562 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-02-10T00:59:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:59:11.562 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-02-10T00:59:11.562 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T00:59:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T00:59:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T00:59:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T00:59:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:11.566 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:11.566 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-02-10T00:59:11.566 INFO:teuthology.orchestra.run.smithi060.stdout:Install 36 Packages 2024-02-10T00:59:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:11.566 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 30 M 2024-02-10T00:59:11.567 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 107 M 2024-02-10T00:59:11.567 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:59:11.967 INFO:teuthology.orchestra.run.smithi060.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-02-10T00:59:12.525 INFO:teuthology.orchestra.run.smithi060.stdout:(2/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-02-10T00:59:12.742 INFO:teuthology.orchestra.run.smithi060.stdout:(3/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 2.2 MB/s | 2.1 MB 00:00 2024-02-10T00:59:12.895 INFO:teuthology.orchestra.run.smithi060.stdout:(4/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86_ 5.1 MB/s | 4.7 MB 00:00 2024-02-10T00:59:12.917 INFO:teuthology.orchestra.run.smithi060.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1361.g2902 1.4 MB/s | 247 kB 00:00 2024-02-10T00:59:13.051 INFO:teuthology.orchestra.run.smithi060.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-02-10T00:59:13.076 INFO:teuthology.orchestra.run.smithi060.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 910 kB/s | 144 kB 00:00 2024-02-10T00:59:13.101 INFO:teuthology.orchestra.run.smithi060.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-02-10T00:59:13.126 INFO:teuthology.orchestra.run.smithi060.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-02-10T00:59:13.152 INFO:teuthology.orchestra.run.smithi060.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-02-10T00:59:13.177 INFO:teuthology.orchestra.run.smithi060.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-10T00:59:13.236 INFO:teuthology.orchestra.run.smithi060.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-02-10T00:59:13.252 INFO:teuthology.orchestra.run.smithi060.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-02-10T00:59:13.286 INFO:teuthology.orchestra.run.smithi060.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-02-10T00:59:13.311 INFO:teuthology.orchestra.run.smithi060.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.9 MB/s | 113 kB 00:00 2024-02-10T00:59:13.337 INFO:teuthology.orchestra.run.smithi060.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-10T00:59:13.454 INFO:teuthology.orchestra.run.smithi060.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-02-10T00:59:13.729 INFO:teuthology.orchestra.run.smithi060.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-02-10T00:59:13.771 INFO:teuthology.orchestra.run.smithi060.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-02-10T00:59:13.913 INFO:teuthology.orchestra.run.smithi060.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 936 kB/s | 172 kB 00:00 2024-02-10T00:59:14.005 INFO:teuthology.orchestra.run.smithi060.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-02-10T00:59:14.030 INFO:teuthology.orchestra.run.smithi060.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-02-10T00:59:14.097 INFO:teuthology.orchestra.run.smithi060.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-02-10T00:59:14.122 INFO:teuthology.orchestra.run.smithi060.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-02-10T00:59:14.315 INFO:teuthology.orchestra.run.smithi060.stdout:(25/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 9.4 MB/s | 17 MB 00:01 2024-02-10T00:59:14.339 INFO:teuthology.orchestra.run.smithi060.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 80 kB/s | 19 kB 00:00 2024-02-10T00:59:14.365 INFO:teuthology.orchestra.run.smithi060.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 81 kB/s | 20 kB 00:00 2024-02-10T00:59:14.465 INFO:teuthology.orchestra.run.smithi060.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-02-10T00:59:14.490 INFO:teuthology.orchestra.run.smithi060.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-02-10T00:59:14.574 INFO:teuthology.orchestra.run.smithi060.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-02-10T00:59:14.599 INFO:teuthology.orchestra.run.smithi060.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 825 kB/s | 90 kB 00:00 2024-02-10T00:59:14.641 INFO:teuthology.orchestra.run.smithi060.stdout:(32/36): python3-logutils-0.3.5-21.el9.noarch.r 142 kB/s | 46 kB 00:00 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /sys 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /proc 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /mnt 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /var/tmp 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /home 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /root 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /tmp 2024-02-10T00:59:14.651 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:14.675 INFO:teuthology.orchestra.run.smithi060.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-02-10T00:59:14.717 INFO:teuthology.orchestra.run.smithi060.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-02-10T00:59:14.775 INFO:teuthology.orchestra.run.smithi060.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-02-10T00:59:15.042 INFO:teuthology.orchestra.run.smithi060.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.0 MB/s | 427 kB 00:00 2024-02-10T00:59:15.046 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:15.046 INFO:teuthology.orchestra.run.smithi060.stdout:Total 8.5 MB/s | 30 MB 00:03 2024-02-10T00:59:15.067 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:15.114 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:15.115 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:15.523 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:15.523 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:15.585 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T00:59:15.609 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T00:59:15.609 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:15.609 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T00:59:15.609 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T00:59:15.609 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T00:59:15.609 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:15.611 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T00:59:15.696 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T00:59:15.696 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T00:59:16.827 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T00:59:16.828 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-10T00:59:16.829 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-10T00:59:16.830 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-10T00:59:16.830 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-10T00:59:16.830 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-10T00:59:16.830 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T00:59:16.830 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-10T00:59:16.830 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-10T00:59:17.007 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:17.079 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-10T00:59:17.111 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-10T00:59:17.164 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-10T00:59:17.255 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-10T00:59:17.313 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-10T00:59:17.346 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-10T00:59:17.441 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-10T00:59:17.528 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-10T00:59:17.574 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-10T00:59:17.658 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-10T00:59:17.709 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-10T00:59:17.746 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-10T00:59:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-10T00:59:17.859 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout:Upgraded: 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.915 INFO:teuthology.orchestra.run.smithi008.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.916 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: re2-1:20211101-3.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:17.917 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:18.054 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-10T00:59:18.117 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-10T00:59:18.160 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-10T00:59:18.198 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-10T00:59:18.228 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-10T00:59:18.257 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-test 2024-02-10T00:59:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-10T00:59:18.338 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T00:59:18.380 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-10T00:59:18.421 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-10T00:59:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-10T00:59:18.507 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-10T00:59:18.599 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-10T00:59:18.638 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-10T00:59:18.714 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-10T00:59:18.844 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:28 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:18.855 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-10T00:59:18.986 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T00:59:19.014 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-10T00:59:19.015 INFO:teuthology.orchestra.run.smithi008.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout:Install 6 Packages 2024-02-10T00:59:19.016 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:19.017 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 48 M 2024-02-10T00:59:19.017 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 204 M 2024-02-10T00:59:19.017 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:59:19.163 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-10T00:59:19.192 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T00:59:19.208 INFO:teuthology.orchestra.run.smithi008.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-02-10T00:59:19.217 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T00:59:19.217 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:19.217 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T00:59:19.217 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T00:59:19.217 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T00:59:19.218 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:19.351 INFO:teuthology.orchestra.run.smithi008.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 442 kB/s | 63 kB 00:00 2024-02-10T00:59:19.376 INFO:teuthology.orchestra.run.smithi008.stdout:(3/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 455 kB/s | 163 kB 00:00 2024-02-10T00:59:19.401 INFO:teuthology.orchestra.run.smithi008.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-10T00:59:19.426 INFO:teuthology.orchestra.run.smithi008.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-10T00:59:20.743 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T00:59:20.769 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T00:59:20.769 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:20.769 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T00:59:20.769 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T00:59:20.769 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T00:59:20.770 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T00:59:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T00:59:21.258 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:21.259 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T00:59:21.259 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T00:59:21.259 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T00:59:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:21.478 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T00:59:21.500 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T00:59:21.501 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:21.501 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T00:59:21.501 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T00:59:21.501 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T00:59:21.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:21.570 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T00:59:22.989 INFO:teuthology.orchestra.run.smithi008.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 12 MB/s | 47 MB 00:03 2024-02-10T00:59:22.993 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:22.993 INFO:teuthology.orchestra.run.smithi008.stdout:Total 12 MB/s | 48 MB 00:03 2024-02-10T00:59:22.993 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:59:23.016 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:59:23.016 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:59:23.194 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-10T00:59:23.195 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-10T00:59:23.196 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T00:59:23.197 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-10T00:59:23.198 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-10T00:59:23.198 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-10T00:59:23.198 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-10T00:59:23.198 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-10T00:59:23.431 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:59:23.432 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:59:24.024 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:59:24.107 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T00:59:24.149 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T00:59:24.192 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T00:59:24.241 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T00:59:24.267 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T00:59:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-10T00:59:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:24.380 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T00:59:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T00:59:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:24.383 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:24.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2024-02-10T00:59:25.337 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:55 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:25.448 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T00:59:25.503 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:25.504 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T00:59:25.504 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:25.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2024-02-10T00:59:26.156 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:56 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:26.321 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:26.322 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:26.323 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:59:26.323 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T00:59:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:26.323 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 748 k 2024-02-10T00:59:26.323 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 754 k 2024-02-10T00:59:26.323 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:59:26.702 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 1.9 MB/s | 748 kB 00:00 2024-02-10T00:59:26.702 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:26.702 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.9 MB/s | 748 kB 00:00 2024-02-10T00:59:26.703 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:26.708 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:26.708 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:26.714 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:26.714 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:27.468 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:27.849 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:28.174 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:28.174 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:28.174 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:28.174 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:28.174 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:28.174 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:28.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T00:59:28.982 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:58 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:28.996 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T00:59:29.078 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T00:59:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:29.148 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T00:59:29.148 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T00:59:29.148 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:29.148 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 145 k 2024-02-10T00:59:29.148 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 436 k 2024-02-10T00:59:29.148 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:59:29.366 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 667 kB/s | 145 kB 00:00 2024-02-10T00:59:29.367 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:29.367 INFO:teuthology.orchestra.run.smithi060.stdout:Total 663 kB/s | 145 kB 00:00 2024-02-10T00:59:29.367 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:29.375 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:29.375 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:29.429 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:29.429 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T00:59:29.741 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T00:59:29.742 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:29.742 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T00:59:29.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:30.740 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T00:59:30.740 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:30.741 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:30.741 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:30.741 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:30.741 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:30.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2024-02-10T00:59:31.482 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T00:59:31.483 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-10T00:59:31.483 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-10T00:59:31.483 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T00:59:31.483 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T00:59:31.483 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T00:59:31.584 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:01 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:31.693 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T00:59:31.748 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:31.749 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T00:59:31.749 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:31.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T00:59:31.851 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: jq-1.6-16.el9.x86_64 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:31.852 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:32.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph 2024-02-10T00:59:32.418 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:02 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:32.585 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-10T00:59:32.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-02-10T00:59:32.587 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:32.588 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 6.8 M 2024-02-10T00:59:32.588 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-02-10T00:59:32.588 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:42 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:32.933 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout:========================================================================================= 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout:========================================================================================= 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-10T00:59:32.936 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T00:59:32.937 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T00:59:32.938 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout:========================================================================================= 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout:Install 36 Packages 2024-02-10T00:59:32.939 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:32.940 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 30 M 2024-02-10T00:59:32.940 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 107 M 2024-02-10T00:59:32.940 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:59:33.503 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1361.g29 105 kB/s | 28 kB 00:00 2024-02-10T00:59:33.537 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1361.g290 50 kB/s | 15 kB 00:00 2024-02-10T00:59:33.684 INFO:teuthology.orchestra.run.smithi008.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-02-10T00:59:33.779 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-02-10T00:59:33.837 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-02-10T00:59:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 2.7 MB/s | 1.8 MB 00:00 2024-02-10T00:59:34.117 INFO:teuthology.orchestra.run.smithi008.stdout:(2/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 3.2 MB/s | 2.1 MB 00:00 2024-02-10T00:59:34.159 INFO:teuthology.orchestra.run.smithi008.stdout:(3/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 2.1 MB/s | 1.5 MB 00:00 2024-02-10T00:59:34.251 INFO:teuthology.orchestra.run.smithi008.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1361.g2902 2.6 MB/s | 247 kB 00:00 2024-02-10T00:59:34.446 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.3 MB/s | 863 kB 00:00 2024-02-10T00:59:34.472 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 33 kB/s | 19 kB 00:00 2024-02-10T00:59:34.476 INFO:teuthology.orchestra.run.smithi008.stdout:(5/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86_ 5.9 MB/s | 4.7 MB 00:00 2024-02-10T00:59:34.518 INFO:teuthology.orchestra.run.smithi008.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 699 kB/s | 187 kB 00:00 2024-02-10T00:59:34.552 INFO:teuthology.orchestra.run.smithi008.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-02-10T00:59:34.577 INFO:teuthology.orchestra.run.smithi008.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 541 kB/s | 32 kB 00:00 2024-02-10T00:59:34.611 INFO:teuthology.orchestra.run.smithi008.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-02-10T00:59:34.636 INFO:teuthology.orchestra.run.smithi008.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 661 kB/s | 39 kB 00:00 2024-02-10T00:59:34.678 INFO:teuthology.orchestra.run.smithi008.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-02-10T00:59:34.787 INFO:teuthology.orchestra.run.smithi008.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-02-10T00:59:34.812 INFO:teuthology.orchestra.run.smithi008.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 772 kB/s | 103 kB 00:00 2024-02-10T00:59:34.846 INFO:teuthology.orchestra.run.smithi008.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-02-10T00:59:34.879 INFO:teuthology.orchestra.run.smithi008.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.7 MB/s | 113 kB 00:00 2024-02-10T00:59:34.905 INFO:teuthology.orchestra.run.smithi008.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-02-10T00:59:34.913 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 327 kB/s | 144 kB 00:00 2024-02-10T00:59:35.039 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-02-10T00:59:35.122 INFO:teuthology.orchestra.run.smithi008.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-02-10T00:59:35.289 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 378 kB/s | 548 kB 00:01 2024-02-10T00:59:35.372 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 565 kB/s | 188 kB 00:00 2024-02-10T00:59:35.555 INFO:teuthology.orchestra.run.smithi008.stdout:(18/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 12 MB/s | 17 MB 00:01 2024-02-10T00:59:35.572 INFO:teuthology.orchestra.run.smithi008.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 97 kB/s | 43 kB 00:00 2024-02-10T00:59:35.597 INFO:teuthology.orchestra.run.smithi008.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 218 kB/s | 151 kB 00:00 2024-02-10T00:59:35.689 INFO:teuthology.orchestra.run.smithi008.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-02-10T00:59:35.790 INFO:teuthology.orchestra.run.smithi008.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-02-10T00:59:35.898 INFO:teuthology.orchestra.run.smithi008.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 214 kB/s | 23 kB 00:00 2024-02-10T00:59:35.949 INFO:teuthology.orchestra.run.smithi008.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 953 kB/s | 358 kB 00:00 2024-02-10T00:59:35.991 INFO:teuthology.orchestra.run.smithi008.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-02-10T00:59:36.041 INFO:teuthology.orchestra.run.smithi008.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-02-10T00:59:36.099 INFO:teuthology.orchestra.run.smithi008.stdout:(27/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 316 kB/s | 172 kB 00:00 2024-02-10T00:59:36.116 INFO:teuthology.orchestra.run.smithi008.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2024-02-10T00:59:36.150 INFO:teuthology.orchestra.run.smithi008.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 725 kB/s | 79 kB 00:00 2024-02-10T00:59:36.233 INFO:teuthology.orchestra.run.smithi008.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 141 kB/s | 16 kB 00:00 2024-02-10T00:59:36.259 INFO:teuthology.orchestra.run.smithi008.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 828 kB/s | 90 kB 00:00 2024-02-10T00:59:36.292 INFO:teuthology.orchestra.run.smithi008.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-02-10T00:59:36.335 INFO:teuthology.orchestra.run.smithi008.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-02-10T00:59:36.394 INFO:teuthology.orchestra.run.smithi008.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-02-10T00:59:36.435 INFO:teuthology.orchestra.run.smithi008.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-02-10T00:59:36.468 INFO:teuthology.orchestra.run.smithi008.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-02-10T00:59:36.474 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:36.474 INFO:teuthology.orchestra.run.smithi008.stdout:Total 8.4 MB/s | 30 MB 00:03 2024-02-10T00:59:36.489 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:59:36.536 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:59:36.537 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:59:36.980 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:59:36.981 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:59:38.473 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:59:38.538 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-10T00:59:38.576 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-10T00:59:38.607 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-10T00:59:38.681 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-10T00:59:38.723 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-10T00:59:38.756 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-10T00:59:38.806 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 479 kB/s | 2.0 MB 00:04 2024-02-10T00:59:38.812 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:38.812 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.1 MB/s | 6.8 MB 00:06 2024-02-10T00:59:38.832 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:38.851 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-10T00:59:38.853 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:38.853 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:38.937 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-10T00:59:39.002 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-10T00:59:39.034 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:39.034 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:39.075 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-10T00:59:39.127 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-10T00:59:39.173 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-10T00:59:39.231 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-10T00:59:39.295 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-10T00:59:39.489 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-10T00:59:39.536 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-10T00:59:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:39.582 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-10T00:59:39.617 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-10T00:59:39.640 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-10T00:59:39.646 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-10T00:59:39.683 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-10T00:59:39.726 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-10T00:59:39.731 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-10T00:59:39.757 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T00:59:39.769 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-10T00:59:39.790 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-10T00:59:39.823 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-10T00:59:39.860 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-10T00:59:39.866 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-10T00:59:39.908 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-10T00:59:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-10T00:59:40.002 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-10T00:59:40.040 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-10T00:59:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-10T00:59:40.116 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-10T00:59:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-10T00:59:40.168 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-10T00:59:40.199 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-10T00:59:40.256 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-10T00:59:40.388 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T00:59:40.454 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-10T00:59:40.491 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T00:59:40.575 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-10T00:59:40.602 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T00:59:40.626 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T00:59:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T00:59:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T00:59:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T00:59:40.627 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:41.314 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T00:59:41.314 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-10T00:59:41.315 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-10T00:59:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-10T00:59:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:41.684 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-10T00:59:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-10T00:59:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T00:59:41.686 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T00:59:41.686 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:41.686 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:41.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T00:59:42.136 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T00:59:42.162 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T00:59:42.162 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:42.162 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T00:59:42.162 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T00:59:42.163 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T00:59:42.163 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:42.544 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:12 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:42.629 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T00:59:42.652 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T00:59:42.652 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:42.652 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T00:59:42.652 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T00:59:42.652 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T00:59:42.652 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:42.712 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:42.713 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-02-10T00:59:42.713 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:59:42.713 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-02-10T00:59:42.713 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T00:59:42.714 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 37 M 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 209 M 2024-02-10T00:59:42.715 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:59:42.879 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T00:59:42.902 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T00:59:42.903 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:42.903 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T00:59:42.903 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T00:59:42.903 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T00:59:42.903 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:42.963 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-02-10T00:59:42.971 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T00:59:42.997 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-02-10T00:59:43.039 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 34 kB 00:00 2024-02-10T00:59:43.106 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-02-10T00:59:43.264 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-02-10T00:59:43.340 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-02-10T00:59:43.365 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-02-10T00:59:43.540 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 802 kB 00:00 2024-02-10T00:59:43.707 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-02-10T00:59:43.732 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-10T00:59:44.241 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.2 MB/s | 7.4 MB 00:01 2024-02-10T00:59:44.691 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 250 kB/s | 245 kB 00:00 2024-02-10T00:59:44.694 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:44.694 INFO:teuthology.orchestra.run.smithi060.stdout:Total 19 MB/s | 37 MB 00:01 2024-02-10T00:59:44.709 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:44.738 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:44.738 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-10T00:59:44.813 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-10T00:59:44.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-10T00:59:44.815 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-10T00:59:44.816 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-10T00:59:45.119 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:45.120 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:45.756 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-10T00:59:45.756 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:45.756 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:59:45.756 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: lua-5.4.4-4.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T00:59:45.757 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T00:59:45.758 INFO:teuthology.orchestra.run.smithi008.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T00:59:45.759 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:45.760 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:45.839 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:45.915 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T00:59:45.946 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T00:59:46.043 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T00:59:46.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-base 2024-02-10T00:59:46.082 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T00:59:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T00:59:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T00:59:46.377 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T00:59:46.667 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:56 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:46.778 INFO:teuthology.orchestra.run.smithi008.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T00:59:46.833 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:46.834 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T00:59:46.834 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:46.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install cephadm 2024-02-10T00:59:47.082 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T00:59:47.198 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T00:59:47.503 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:57 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:47.670 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 748 k 2024-02-10T00:59:47.671 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 754 k 2024-02-10T00:59:47.672 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:59:48.095 INFO:teuthology.orchestra.run.smithi008.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-02-10T00:59:48.096 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:48.096 INFO:teuthology.orchestra.run.smithi008.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-02-10T00:59:48.096 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:59:48.101 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:59:48.102 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:59:48.108 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:59:48.108 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:59:48.306 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:59:48.524 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T00:59:48.746 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:48.828 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:49.203 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:49.559 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T00:59:49.560 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:49.560 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:59:49.560 INFO:teuthology.orchestra.run.smithi008.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:49.560 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:49.560 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:49.623 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T00:59:49.655 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T00:59:49.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T00:59:50.345 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:00 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:50.508 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:50.509 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:59:50.510 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T00:59:50.510 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T00:59:50.510 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:50.510 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 145 k 2024-02-10T00:59:50.510 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 436 k 2024-02-10T00:59:50.510 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:59:50.786 INFO:teuthology.orchestra.run.smithi008.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 527 kB/s | 145 kB 00:00 2024-02-10T00:59:50.787 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:50.787 INFO:teuthology.orchestra.run.smithi008.stdout:Total 524 kB/s | 145 kB 00:00 2024-02-10T00:59:50.787 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:59:50.795 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:59:50.795 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:59:50.850 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:59:50.850 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:59:51.116 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:59:51.187 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T00:59:51.211 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T00:59:51.211 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T00:59:51.211 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T00:59:51.211 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T00:59:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-10T00:59:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T00:59:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T00:59:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T00:59:52.203 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T00:59:52.203 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:52.203 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:59:52.203 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T00:59:52.204 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:52.204 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:52.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T00:59:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T00:59:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T00:59:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T00:59:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T00:59:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T00:59:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:52.647 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:53.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2024-02-10T00:59:53.024 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:53.135 INFO:teuthology.orchestra.run.smithi008.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T00:59:53.188 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:53.190 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T00:59:53.190 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:53.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T00:59:53.643 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:23 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:53.808 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T00:59:53.810 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2024-02-10T00:59:53.811 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:53.812 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.0 M 2024-02-10T00:59:53.812 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-02-10T00:59:53.812 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T00:59:53.863 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:54.028 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout:====================================================================================== 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout:====================================================================================== 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-10T00:59:54.029 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout:====================================================================================== 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout:Install 12 Packages 2024-02-10T00:59:54.030 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:54.031 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 6.8 M 2024-02-10T00:59:54.031 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 26 M 2024-02-10T00:59:54.031 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:59:54.064 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 18 kB 00:00 2024-02-10T00:59:54.081 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 109 kB/s | 24 kB 00:00 2024-02-10T00:59:54.247 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 951 kB/s | 174 kB 00:00 2024-02-10T00:59:54.265 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 128 kB/s | 52 kB 00:00 2024-02-10T00:59:54.290 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 633 kB/s | 132 kB 00:00 2024-02-10T00:59:54.323 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-02-10T00:59:54.349 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-02-10T00:59:54.674 INFO:teuthology.orchestra.run.smithi008.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1361.g29 91 kB/s | 28 kB 00:00 2024-02-10T00:59:54.699 INFO:teuthology.orchestra.run.smithi008.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1361.g290 46 kB/s | 15 kB 00:00 2024-02-10T00:59:54.757 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 35 kB/s | 15 kB 00:00 2024-02-10T00:59:54.807 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 62 kB/s | 32 kB 00:00 2024-02-10T00:59:55.015 INFO:teuthology.orchestra.run.smithi008.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 751 kB/s | 238 kB 00:00 2024-02-10T00:59:55.099 INFO:teuthology.orchestra.run.smithi008.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2024-02-10T00:59:55.124 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 261 kB/s | 203 kB 00:00 2024-02-10T00:59:55.150 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-02-10T00:59:55.200 INFO:teuthology.orchestra.run.smithi008.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 2.1 MB/s | 1.8 MB 00:00 2024-02-10T00:59:55.283 INFO:teuthology.orchestra.run.smithi008.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-02-10T00:59:55.317 INFO:teuthology.orchestra.run.smithi008.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-02-10T00:59:55.367 INFO:teuthology.orchestra.run.smithi008.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-02-10T00:59:55.392 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 336 kB/s | 90 kB 00:00 2024-02-10T00:59:55.434 INFO:teuthology.orchestra.run.smithi008.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 741 kB/s | 31 kB 00:00 2024-02-10T00:59:55.484 INFO:teuthology.orchestra.run.smithi008.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-02-10T00:59:55.492 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-02-10T00:59:55.497 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:55.498 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-02-10T00:59:55.498 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T00:59:55.514 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T00:59:55.514 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T00:59:55.551 INFO:teuthology.orchestra.run.smithi008.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.6 MB/s | 2.0 MB 00:00 2024-02-10T00:59:55.625 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T00:59:55.625 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T00:59:55.776 INFO:teuthology.orchestra.run.smithi008.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-02-10T00:59:55.781 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T00:59:55.781 INFO:teuthology.orchestra.run.smithi008.stdout:Total 3.9 MB/s | 6.8 MB 00:01 2024-02-10T00:59:55.799 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T00:59:55.817 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T00:59:55.817 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T00:59:56.000 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T00:59:56.000 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T00:59:56.158 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T00:59:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T00:59:56.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T00:59:56.450 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T00:59:56.486 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T00:59:56.527 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T00:59:56.546 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T00:59:56.625 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T00:59:56.636 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-10T00:59:56.671 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-10T00:59:56.718 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-10T00:59:56.734 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T00:59:56.748 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-10T00:59:56.796 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T00:59:56.834 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-10T00:59:56.970 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-10T00:59:57.070 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-10T00:59:57.120 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-10T00:59:57.147 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-10T00:59:57.178 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-10T00:59:57.432 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-10T00:59:57.473 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T00:59:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T00:59:57.618 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T00:59:57.651 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T00:59:57.701 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T00:59:57.729 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-10T00:59:58.428 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-10T00:59:58.429 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-10T00:59:58.429 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-10T00:59:58.429 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-10T00:59:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T00:59:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T00:59:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T00:59:58.795 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-10T00:59:58.795 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:58.795 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T00:59:58.795 INFO:teuthology.orchestra.run.smithi008.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-10T00:59:58.795 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:58.795 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:58.796 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T00:59:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T00:59:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:58.869 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T00:59:59.043 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T00:59:59.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T00:59:59.635 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:09 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T00:59:59.699 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:29 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T00:59:59.803 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================================ 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================================ 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T00:59:59.804 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================================ 2024-02-10T00:59:59.805 INFO:teuthology.orchestra.run.smithi008.stdout:Install 12 Packages 2024-02-10T00:59:59.806 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T00:59:59.806 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 37 M 2024-02-10T00:59:59.806 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 209 M 2024-02-10T00:59:59.806 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T00:59:59.869 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T00:59:59.869 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T00:59:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T00:59:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:59.871 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T00:59:59.871 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-02-10T00:59:59.871 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-02-10T00:59:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T00:59:59.871 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 6.8 M 2024-02-10T00:59:59.872 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2024-02-10T00:59:59.872 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:00.118 INFO:teuthology.orchestra.run.smithi008.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 32 kB 00:00 2024-02-10T01:00:00.144 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 996 kB/s | 227 kB 00:00 2024-02-10T01:00:00.168 INFO:teuthology.orchestra.run.smithi008.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 349 kB/s | 17 kB 00:00 2024-02-10T01:00:00.235 INFO:teuthology.orchestra.run.smithi008.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 512 kB/s | 34 kB 00:00 2024-02-10T01:00:00.319 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-02-10T01:00:00.361 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 310 kB/s | 138 kB 00:00 2024-02-10T01:00:00.528 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 278 kB/s | 58 kB 00:00 2024-02-10T01:00:00.553 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 266 kB/s | 51 kB 00:00 2024-02-10T01:00:00.628 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-02-10T01:00:00.632 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:00.633 INFO:teuthology.orchestra.run.smithi060.stdout:Total 8.9 MB/s | 6.8 MB 00:00 2024-02-10T01:00:00.633 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:00.643 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:00.643 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:00.740 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:00.740 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:01.039 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:01.161 INFO:teuthology.orchestra.run.smithi008.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-02-10T01:00:01.218 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T01:00:01.273 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T01:00:01.825 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T01:00:01.911 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T01:00:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T01:00:02.009 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T01:00:02.094 INFO:teuthology.orchestra.run.smithi008.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.4 MB/s | 3.0 MB 00:02 2024-02-10T01:00:02.461 INFO:teuthology.orchestra.run.smithi008.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.7 MB/s | 4.8 MB 00:01 2024-02-10T01:00:02.486 INFO:teuthology.orchestra.run.smithi008.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 555 kB/s | 217 kB 00:00 2024-02-10T01:00:02.778 INFO:teuthology.orchestra.run.smithi008.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 802 kB 00:00 2024-02-10T01:00:02.818 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T01:00:02.818 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-10T01:00:02.818 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T01:00:02.819 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T01:00:02.819 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T01:00:02.819 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T01:00:03.020 INFO:teuthology.orchestra.run.smithi008.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 809 kB/s | 194 kB 00:00 2024-02-10T01:00:03.129 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T01:00:03.129 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:03.129 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:03.129 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T01:00:03.129 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T01:00:03.129 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T01:00:03.130 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T01:00:03.130 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T01:00:03.130 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T01:00:03.130 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:03.130 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:03.279 INFO:teuthology.orchestra.run.smithi008.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-02-10T01:00:03.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2024-02-10T01:00:03.537 INFO:teuthology.orchestra.run.smithi008.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 2.1 MB/s | 7.4 MB 00:03 2024-02-10T01:00:03.754 INFO:teuthology.orchestra.run.smithi008.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 335 kB/s | 245 kB 00:00 2024-02-10T01:00:03.758 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:03.758 INFO:teuthology.orchestra.run.smithi008.stdout:Total 9.4 MB/s | 37 MB 00:03 2024-02-10T01:00:03.772 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:03.799 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:03.799 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:03.962 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:04.127 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:04.127 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:04.127 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-02-10T01:00:04.128 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:04.129 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 964 k 2024-02-10T01:00:04.129 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.8 M 2024-02-10T01:00:04.129 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:04.164 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:04.165 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:04.263 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 586 kB/s | 79 kB 00:00 2024-02-10T01:00:04.463 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 2.6 MB/s | 885 kB 00:00 2024-02-10T01:00:04.464 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:04.465 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-02-10T01:00:04.465 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:04.487 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:04.487 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:04.566 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:04.566 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:04.761 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:04.902 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T01:00:04.974 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T01:00:05.009 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:05.069 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T01:00:05.109 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T01:00:05.206 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T01:00:05.262 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T01:00:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T01:00:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T01:00:05.556 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T01:00:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T01:00:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-10T01:00:06.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T01:00:06.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:06.062 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:06.062 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T01:00:06.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:06.062 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:06.245 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T01:00:06.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2024-02-10T01:00:06.360 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T01:00:06.921 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-10T01:00:07.086 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 332 k 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.5 M 2024-02-10T01:00:07.087 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:07.212 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 550 kB/s | 69 kB 00:00 2024-02-10T01:00:07.354 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 989 kB/s | 264 kB 00:00 2024-02-10T01:00:07.355 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:07.355 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-02-10T01:00:07.355 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:07.364 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:07.365 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:07.408 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:07.409 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:07.568 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:07.676 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T01:00:07.768 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-10T01:00:07.806 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T01:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T01:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T01:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-10T01:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:08.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-10T01:00:08.765 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T01:00:08.794 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T01:00:08.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-10T01:00:08.955 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:08.955 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:08.956 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T01:00:08.956 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-10T01:00:08.956 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:08.956 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:09.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2024-02-10T01:00:09.875 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:10.043 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:10.044 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:10.044 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-02-10T01:00:10.044 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:10.044 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:10.044 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 127 k 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 456 k 2024-02-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:10.269 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 567 kB/s | 127 kB 00:00 2024-02-10T01:00:10.269 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:10.270 INFO:teuthology.orchestra.run.smithi060.stdout:Total 563 kB/s | 127 kB 00:00 2024-02-10T01:00:10.270 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:10.276 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:10.276 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:10.333 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:10.334 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:10.492 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:10.587 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:11.045 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:11.346 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:11.347 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T01:00:11.381 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T01:00:11.382 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T01:00:11.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2024-02-10T01:00:11.896 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T01:00:11.897 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:11.897 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:11.897 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T01:00:11.897 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T01:00:11.897 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T01:00:11.897 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T01:00:11.898 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T01:00:11.899 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T01:00:11.899 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:11.899 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:12.201 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:42 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:12.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-rook 2024-02-10T01:00:12.312 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:12.367 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:12.368 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:12.368 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:12.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2024-02-10T01:00:12.856 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:22 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:13.021 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T01:00:13.023 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T01:00:13.024 INFO:teuthology.orchestra.run.smithi008.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout:Install 13 Packages 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 2.0 M 2024-02-10T01:00:13.025 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 26 M 2024-02-10T01:00:13.026 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:13.028 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:13.195 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:13.196 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:13.196 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 31 k 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 140 k 2024-02-10T01:00:13.197 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:13.260 INFO:teuthology.orchestra.run.smithi008.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 154 kB/s | 24 kB 00:00 2024-02-10T01:00:13.285 INFO:teuthology.orchestra.run.smithi008.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 98 kB/s | 18 kB 00:00 2024-02-10T01:00:13.328 INFO:teuthology.orchestra.run.smithi008.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-02-10T01:00:13.353 INFO:teuthology.orchestra.run.smithi008.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-02-10T01:00:13.376 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 178 kB/s | 31 kB 00:00 2024-02-10T01:00:13.376 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:13.376 INFO:teuthology.orchestra.run.smithi060.stdout:Total 176 kB/s | 31 kB 00:00 2024-02-10T01:00:13.376 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:13.377 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:13.377 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:13.378 INFO:teuthology.orchestra.run.smithi008.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T01:00:13.403 INFO:teuthology.orchestra.run.smithi008.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-02-10T01:00:13.411 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:13.412 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:13.429 INFO:teuthology.orchestra.run.smithi008.stdout:(7/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 160 kB/s | 52 kB 00:00 2024-02-10T01:00:13.487 INFO:teuthology.orchestra.run.smithi008.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-02-10T01:00:13.513 INFO:teuthology.orchestra.run.smithi008.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-02-10T01:00:13.535 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:13.571 INFO:teuthology.orchestra.run.smithi008.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-02-10T01:00:13.596 INFO:teuthology.orchestra.run.smithi008.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-02-10T01:00:13.628 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:13.630 INFO:teuthology.orchestra.run.smithi008.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-02-10T01:00:13.764 INFO:teuthology.orchestra.run.smithi008.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-02-10T01:00:13.765 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:13.766 INFO:teuthology.orchestra.run.smithi008.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-02-10T01:00:13.766 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:13.780 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:13.780 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:13.882 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:13.883 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:14.039 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:14.345 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:14.345 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:14.346 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:14.346 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:14.346 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:14.346 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:14.422 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:14.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2024-02-10T01:00:14.584 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T01:00:14.642 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T01:00:14.698 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T01:00:14.733 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T01:00:14.775 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T01:00:14.873 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T01:00:14.992 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T01:00:15.044 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T01:00:15.108 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:15.217 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:15.271 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:15.272 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:15.272 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:15.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2024-02-10T01:00:15.821 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T01:00:15.881 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T01:00:15.915 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:15.916 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T01:00:15.964 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T01:00:15.993 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T01:00:16.025 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:16.079 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:16.080 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:16.080 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:16.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2024-02-10T01:00:16.729 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:16.806 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T01:00:16.806 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-10T01:00:16.807 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T01:00:16.807 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T01:00:16.807 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T01:00:16.807 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T01:00:16.807 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T01:00:16.807 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T01:00:16.808 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T01:00:16.808 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T01:00:16.808 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T01:00:16.808 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T01:00:16.809 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T01:00:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:16.894 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:16.895 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:16.895 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:16.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2024-02-10T01:00:17.177 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T01:00:17.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T01:00:17.179 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:17.179 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:17.433 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T01:00:17.545 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:17.655 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:17.709 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:17.710 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:17.710 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:17.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2024-02-10T01:00:18.025 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:28 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:18.193 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T01:00:18.194 INFO:teuthology.orchestra.run.smithi008.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout:Install 6 Packages 2024-02-10T01:00:18.195 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:18.196 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 6.8 M 2024-02-10T01:00:18.196 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 33 M 2024-02-10T01:00:18.196 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:18.358 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:48 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:18.469 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:18.524 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:18.525 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:18.526 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:18.559 INFO:teuthology.orchestra.run.smithi008.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-02-10T01:00:18.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2024-02-10T01:00:18.708 INFO:teuthology.orchestra.run.smithi008.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-02-10T01:00:18.758 INFO:teuthology.orchestra.run.smithi008.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 374 kB/s | 138 kB 00:00 2024-02-10T01:00:18.917 INFO:teuthology.orchestra.run.smithi008.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2024-02-10T01:00:18.942 INFO:teuthology.orchestra.run.smithi008.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-02-10T01:00:18.967 INFO:teuthology.orchestra.run.smithi008.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 244 kB/s | 51 kB 00:00 2024-02-10T01:00:18.969 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:18.969 INFO:teuthology.orchestra.run.smithi008.stdout:Total 8.7 MB/s | 6.8 MB 00:00 2024-02-10T01:00:18.969 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:18.979 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:18.979 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:19.073 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:19.074 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:19.171 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:49 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:19.281 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:19.334 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:19.335 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:19.335 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:19.384 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:19.392 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2024-02-10T01:00:19.549 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T01:00:19.612 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T01:00:19.986 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:49 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:20.150 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:20.151 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:20.151 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-02-10T01:00:20.151 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:20.151 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:20.151 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-10T01:00:20.151 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 85 k 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 231 k 2024-02-10T01:00:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:20.163 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T01:00:20.250 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T01:00:20.319 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T01:00:20.348 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T01:00:20.401 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 342 kB/s | 85 kB 00:00 2024-02-10T01:00:20.402 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:20.402 INFO:teuthology.orchestra.run.smithi060.stdout:Total 341 kB/s | 85 kB 00:00 2024-02-10T01:00:20.402 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:20.409 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:20.409 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:20.466 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:20.466 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:20.607 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:20.701 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:21.095 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:21.202 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T01:00:21.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-10T01:00:21.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T01:00:21.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T01:00:21.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T01:00:21.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T01:00:21.343 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:21.344 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:21.344 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:21.344 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:21.344 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:21.344 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:21.524 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T01:00:21.524 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:21.525 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:21.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-02-10T01:00:21.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-fuse 2024-02-10T01:00:22.171 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:52 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.1 M 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 13 M 2024-02-10T01:00:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:22.358 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:32 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:22.525 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:22.526 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:22.526 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T01:00:22.526 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Install 2 Packages 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 964 k 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 2.8 M 2024-02-10T01:00:22.527 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:22.660 INFO:teuthology.orchestra.run.smithi008.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 593 kB/s | 79 kB 00:00 2024-02-10T01:00:22.845 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 6.1 MB/s | 3.1 MB 00:00 2024-02-10T01:00:22.846 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:22.846 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.1 MB/s | 3.1 MB 00:00 2024-02-10T01:00:22.847 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:22.856 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:22.856 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:22.877 INFO:teuthology.orchestra.run.smithi008.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-02-10T01:00:22.878 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:22.878 INFO:teuthology.orchestra.run.smithi008.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-02-10T01:00:22.879 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:22.900 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:22.900 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:22.932 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:22.933 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:22.978 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:22.979 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:23.172 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:23.307 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T01:00:23.370 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:23.380 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T01:00:23.430 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:23.454 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:23.454 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T01:00:23.454 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T01:00:23.454 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T01:00:23.455 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T01:00:23.455 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:24.060 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T01:00:24.060 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-10T01:00:24.404 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T01:00:24.404 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:24.405 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:24.405 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T01:00:24.405 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:24.405 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:24.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:24.610 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:24.610 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:24.610 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:24.610 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:24.610 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:24.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-volume 2024-02-10T01:00:24.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2024-02-10T01:00:25.278 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:35 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:25.443 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout:Install 2 Packages 2024-02-10T01:00:25.444 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:25.445 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 332 k 2024-02-10T01:00:25.445 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 1.5 M 2024-02-10T01:00:25.445 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:25.453 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:55 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:25.576 INFO:teuthology.orchestra.run.smithi008.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 526 kB/s | 69 kB 00:00 2024-02-10T01:00:25.618 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:25.618 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 171 k 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 494 k 2024-02-10T01:00:25.619 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-02-10T01:00:25.734 INFO:teuthology.orchestra.run.smithi008.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 912 kB/s | 264 kB 00:00 2024-02-10T01:00:25.735 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:25.735 INFO:teuthology.orchestra.run.smithi008.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-02-10T01:00:25.735 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:25.749 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:25.749 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:25.792 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:25.792 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:25.874 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 671 kB/s | 171 kB 00:00 2024-02-10T01:00:25.875 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:25.875 INFO:teuthology.orchestra.run.smithi060.stdout:Total 667 kB/s | 171 kB 00:00 2024-02-10T01:00:25.875 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-02-10T01:00:25.883 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-02-10T01:00:25.883 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-02-10T01:00:25.941 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-02-10T01:00:25.941 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-02-10T01:00:25.953 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:26.099 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-02-10T01:00:26.144 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-10T01:00:26.178 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T01:00:26.194 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T01:00:26.194 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T01:00:26.194 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-10T01:00:26.194 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:26.200 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:26.624 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:26.908 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-10T01:00:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:26.959 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-02-10T01:00:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:26.959 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:27.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-02-10T01:00:27.265 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-10T01:00:27.265 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:27.266 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:27.266 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T01:00:27.266 INFO:teuthology.orchestra.run.smithi008.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-10T01:00:27.266 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:27.266 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:27.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install librados-devel 2024-02-10T01:00:27.764 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:57 ago on Sat 10 Feb 2024 12:58:30 AM UTC. 2024-02-10T01:00:27.874 INFO:teuthology.orchestra.run.smithi060.stdout:Package rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:27.928 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-02-10T01:00:27.929 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-02-10T01:00:27.929 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-02-10T01:00:28.001 DEBUG:teuthology.parallel:result is None 2024-02-10T01:00:28.226 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:38 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:28.397 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:28.398 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:28.398 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T01:00:28.398 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 127 k 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 456 k 2024-02-10T01:00:28.399 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:28.649 INFO:teuthology.orchestra.run.smithi008.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 508 kB/s | 127 kB 00:00 2024-02-10T01:00:28.649 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:28.650 INFO:teuthology.orchestra.run.smithi008.stdout:Total 504 kB/s | 127 kB 00:00 2024-02-10T01:00:28.650 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:28.656 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:28.656 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:28.714 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:28.714 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:28.870 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:28.984 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:29.436 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:29.748 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:29.748 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:29.749 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:29.749 INFO:teuthology.orchestra.run.smithi008.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:29.749 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:29.749 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:29.988 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install libcephfs2 2024-02-10T01:00:30.592 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:30.703 INFO:teuthology.orchestra.run.smithi008.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:30.759 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:30.760 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:30.760 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:30.837 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install libcephfs-devel 2024-02-10T01:00:31.432 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:41 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:31.604 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:31.604 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 31 k 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 140 k 2024-02-10T01:00:31.605 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:31.877 INFO:teuthology.orchestra.run.smithi008.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 115 kB/s | 31 kB 00:00 2024-02-10T01:00:31.878 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:31.878 INFO:teuthology.orchestra.run.smithi008.stdout:Total 114 kB/s | 31 kB 00:00 2024-02-10T01:00:31.878 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:31.882 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:31.882 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:31.901 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:31.901 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:32.046 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:32.146 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:32.533 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:32.814 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:32.814 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:32.814 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:32.814 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:32.815 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:32.815 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:32.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install librados2 2024-02-10T01:00:33.574 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:33.682 INFO:teuthology.orchestra.run.smithi008.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:33.736 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:33.737 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:33.737 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:33.780 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install librbd1 2024-02-10T01:00:34.372 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:34.480 INFO:teuthology.orchestra.run.smithi008.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:34.534 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:34.535 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:34.536 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:34.580 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-rados 2024-02-10T01:00:35.179 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:35.290 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:35.344 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:35.345 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:35.345 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:35.400 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-rgw 2024-02-10T01:00:36.001 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:36.110 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:36.163 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:36.165 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:36.165 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:36.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-cephfs 2024-02-10T01:00:36.802 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:36.912 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:36.967 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:36.968 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:36.968 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:37.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-rbd 2024-02-10T01:00:37.609 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:37.719 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:37.773 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:37.774 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:37.774 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:37.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-fuse 2024-02-10T01:00:38.417 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:48 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:38.582 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:38.582 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:38.582 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-02-10T01:00:38.582 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 85 k 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 231 k 2024-02-10T01:00:38.583 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:38.864 INFO:teuthology.orchestra.run.smithi008.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 304 kB/s | 85 kB 00:00 2024-02-10T01:00:38.864 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:38.865 INFO:teuthology.orchestra.run.smithi008.stdout:Total 302 kB/s | 85 kB 00:00 2024-02-10T01:00:38.865 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:38.873 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:38.873 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:38.930 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:38.930 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:39.073 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:39.181 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:39.583 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:39.879 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:39.880 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:39.880 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:39.880 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:39.880 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:40.132 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-mirror 2024-02-10T01:00:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:50 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:40.899 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T01:00:40.900 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:40.901 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 3.1 M 2024-02-10T01:00:40.901 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 13 M 2024-02-10T01:00:40.901 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:41.959 INFO:teuthology.orchestra.run.smithi008.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-02-10T01:00:41.960 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:41.960 INFO:teuthology.orchestra.run.smithi008.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-02-10T01:00:41.960 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:41.971 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:41.972 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:42.049 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:42.050 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:42.489 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:42.560 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:42.583 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:42.583 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T01:00:42.584 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T01:00:42.584 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T01:00:42.584 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T01:00:42.584 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:43.712 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:43.712 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:43.713 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:43.982 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-nbd 2024-02-10T01:00:44.583 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:54 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:44.745 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T01:00:44.746 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T01:00:44.747 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T01:00:44.747 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:44.747 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 171 k 2024-02-10T01:00:44.747 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 494 k 2024-02-10T01:00:44.747 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T01:00:45.072 INFO:teuthology.orchestra.run.smithi008.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 528 kB/s | 171 kB 00:00 2024-02-10T01:00:45.072 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T01:00:45.072 INFO:teuthology.orchestra.run.smithi008.stdout:Total 526 kB/s | 171 kB 00:00 2024-02-10T01:00:45.072 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T01:00:45.080 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T01:00:45.080 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T01:00:45.137 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T01:00:45.137 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T01:00:45.290 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T01:00:45.389 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:45.787 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:46.105 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T01:00:46.105 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:46.105 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T01:00:46.105 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T01:00:46.105 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:46.105 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:46.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-mirror 2024-02-10T01:00:46.940 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:56 ago on Sat 10 Feb 2024 12:58:50 AM UTC. 2024-02-10T01:00:47.051 INFO:teuthology.orchestra.run.smithi008.stdout:Package rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T01:00:47.108 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T01:00:47.108 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T01:00:47.109 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T01:00:47.169 DEBUG:teuthology.parallel:result is None 2024-02-10T01:00:47.170 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T01:00:47.349 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T01:00:47.370 INFO:teuthology.orchestra.run.smithi008.stdout:19.0.0-1361.g290212a1.el9 2024-02-10T01:00:47.371 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-10T01:00:47.371 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-10T01:00:47.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T01:00:47.529 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T01:00:47.553 INFO:teuthology.orchestra.run.smithi060.stdout:19.0.0-1361.g290212a1.el9 2024-02-10T01:00:47.553 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-10T01:00:47.553 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-10T01:00:47.555 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T01:00:47.555 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:47.555 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T01:00:47.584 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:00:47.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T01:00:47.624 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T01:00:47.624 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:47.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T01:00:47.652 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T01:00:47.720 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:00:47.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T01:00:47.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T01:00:47.820 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T01:00:47.821 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:47.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T01:00:47.848 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T01:00:47.915 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:00:47.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T01:00:47.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T01:00:48.014 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T01:00:48.014 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:48.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T01:00:48.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T01:00:48.109 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:00:48.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T01:00:48.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T01:00:48.204 INFO:teuthology.run_tasks:Running task ceph... 2024-02-10T01:00:48.291 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-10T01:00:48.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 777 /var/log/ceph 2024-02-10T01:00:48.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-02-10T01:00:48.320 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-10T01:00:48.321 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-10T01:00:48.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-10T01:00:48.388 INFO:tasks.ceph:Creating extra log directories... 2024-02-10T01:00:48.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-10T01:00:48.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-10T01:00:48.469 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-02-10T01:00:48.469 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-10T01:00:48.470 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7554696', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5679, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'targets': {'smithi008.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlSAKYrhH4t6lqydJwPc7dakwrUcEIWsSCqIlLXUX8KcGMzn9wmRKv3395yFBqkRi00GYbssmuqVssZshiM65E=', 'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3fOJ0AL93zBAlc9KUWLcYTG+/ZN22p+OeOUdu5BJmCIqUtDwSyLdMZQynzTw6sRiugO8+IUpLWGgYqOxmkkzs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd9fdb2209e15b39d9f061fd85399f352ce0f0894', 'timestamp': '2024-02-09_19:39:00', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-10T01:00:48.470 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-02-10T01:00:48.495 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-02-10T01:00:48.515 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/run/ceph 2024-02-10T01:00:48.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-02-10T01:00:48.583 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:48.583 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T01:00:48.630 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T01:00:48.630 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 2024-02-10T01:00:48.685 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T01:00:48.685 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:00:48.685 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T01:00:48.685 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:00:48.685 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:00:48.685 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 01:00:42.835030611 +0000 2024-02-10T01:00:48.686 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 00:57:31.446476164 +0000 2024-02-10T01:00:48.686 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 00:57:31.446476164 +0000 2024-02-10T01:00:48.686 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 00:57:31.446476164 +0000 2024-02-10T01:00:48.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T01:00:48.751 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T01:00:48.751 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T01:00:48.751 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000112258 s, 4.6 MB/s 2024-02-10T01:00:48.752 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T01:00:48.808 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 01:00:42.835030611 +0000 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 00:57:31.771473074 +0000 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 00:57:31.771473074 +0000 2024-02-10T01:00:48.864 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 00:57:31.771473074 +0000 2024-02-10T01:00:48.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T01:00:48.931 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T01:00:48.931 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T01:00:48.931 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 9.386e-05 s, 5.5 MB/s 2024-02-10T01:00:48.932 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T01:00:48.988 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 2024-02-10T01:00:49.043 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T01:00:49.043 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:00:49.043 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 01:00:42.836030602 +0000 2024-02-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 00:57:32.110469850 +0000 2024-02-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 00:57:32.110469850 +0000 2024-02-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 00:57:32.110469850 +0000 2024-02-10T01:00:49.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T01:00:49.113 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T01:00:49.113 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T01:00:49.113 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 7.97e-05 s, 6.4 MB/s 2024-02-10T01:00:49.113 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T01:00:49.168 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 2024-02-10T01:00:49.222 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T01:00:49.222 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:00:49.222 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T01:00:49.223 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:00:49.223 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:00:49.223 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 01:00:42.836030602 +0000 2024-02-10T01:00:49.223 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 00:57:32.448466637 +0000 2024-02-10T01:00:49.223 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 00:57:32.448466637 +0000 2024-02-10T01:00:49.223 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 00:57:32.448466637 +0000 2024-02-10T01:00:49.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T01:00:49.287 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T01:00:49.288 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T01:00:49.288 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 8.1424e-05 s, 6.3 MB/s 2024-02-10T01:00:49.289 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T01:00:49.345 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:00:49.345 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi008.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-10T01:00:49.345 INFO:tasks.ceph:Generating config... 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client] debug rbd = 20 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client] log to stderr = False 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client] rbd default features = 125 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.348 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.349 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.349 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-02-10T01:00:49.349 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-02-10T01:00:49.349 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-02-10T01:00:49.349 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-10T01:00:49.349 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-10T01:00:49.349 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-10T01:00:49.349 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-10T01:00:49.349 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-10T01:00:49.349 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-02-10T01:00:49.349 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-02-10T01:00:49.349 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-10T01:00:49.349 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-10T01:00:49.349 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-02-10T01:00:49.349 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-10T01:00:49.350 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-10T01:00:49.350 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-02-10T01:00:49.350 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-02-10T01:00:49.445 INFO:teuthology.orchestra.run.smithi008.stdout:creating /etc/ceph/cluster1.keyring 2024-02-10T01:00:49.448 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-02-10T01:00:49.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-02-10T01:00:49.558 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.8')] 2024-02-10T01:00:49.559 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.8', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-02-10T01:00:49.560 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:49.560 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-10T01:00:49.613 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.8 --print /home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:49.690 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: generated fsid c022c2cc-868d-45e0-bbf6-c3f8e32882f5 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:setting min_mon_release = quincy 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:epoch 0 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:fsid c022c2cc-868d-45e0-bbf6-c3f8e32882f5 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:last_changed 2024-02-10T01:00:49.690538+0000 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:created 2024-02-10T01:00:49.690538+0000 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:min_mon_release 17 (quincy) 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:election_strategy: 1 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.a 2024-02-10T01:00:49.691 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-02-10T01:00:49.692 DEBUG:teuthology.orchestra.run.smithi008:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-10T01:00:49.747 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID c022c2cc-868d-45e0-bbf6-c3f8e32882f5... 2024-02-10T01:00:49.748 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-02-10T01:00:49.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-02-10T01:00:49.837 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: chdir = "" 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: auth supported = cephx 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: filestore xattr use omap = true 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: mon clock drift allowed = 1.000 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: osd crush chooseleaf type = 0 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: auth debug = true 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.838 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on old message = true 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on bug = true 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon pg warn max object skew = 0 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: # disable pg_autoscaler by default for new pools 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default size = 2 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow primary affinity = true 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow pg remap = true 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on legacy crush tunables = false 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on crush straw calc version zero = false 2024-02-10T01:00:49.839 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on no sortbitwise = false 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on osd down out interval zero = false 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on too few osds = false 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: mon_allow_pool_size_one = true 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: osd default data pool replay window = 5 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: mon allow pool delete = true 2024-02-10T01:00:49.840 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: mon cluster log file level = debug 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: debug asserts on shutdown = true 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: mon health detail to clog = false 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: mon host = 172.21.15.8 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: mon client directed command retry = 5 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on pool no app = False 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject socket failures = 5000 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = c022c2cc-868d-45e0-bbf6-c3f8e32882f5 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout:[osd] 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: osd journal size = 100 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub load threshold = 5.0 2024-02-10T01:00:49.841 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub max interval = 600 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd mclock profile = high_recovery_ops 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd recover clone overlap = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd recovery max chunk = 1048576 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug shutdown = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug op order = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug verify stray on activate = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug trim objects = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd open classes on start = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug pg log writeout = true 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: osd deep scrub update digest min age = 30 2024-02-10T01:00:49.842 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: osd map max advance = 10 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: journal zero on create = true 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: filestore ondisk finisher threads = 3 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: filestore apply finisher threads = 3 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: bdev debug aio = true 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug misdirected ops = true 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: bluefs allocator = hybrid 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore allocator = hybrid 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore block size = 96636764160 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore fsck on mount = True 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluefs = 1/20 2024-02-10T01:00:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluestore = 1/20 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: debug osd = 20 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: debug rocksdb = 4/10 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd backfillfull_ratio = 0.85 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd full ratio = 0.9 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd nearfull ratio = 0.8 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: osd failsafe full ratio = 0.95 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: osd objectstore = bluestore 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr] 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: debug mgr = 20 2024-02-10T01:00:49.844 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mgr/telemetry/nag = false 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout:[mon] 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: debug paxos = 20 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon data avail warn = 5 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon mgr mkfs grace = 240 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd reporter subtree level = osd 2024-02-10T01:00:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd prime pg temp = true 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: auth mon ticket ttl = 660 # 11m 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: auth service ticket ttl = 240 # 4m 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: # don't complain about insecure global_id in the test suite 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: # 1m isn't quite enough 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: mon_down_mkfs_grace = 2m 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.846 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_filestore_osds = false 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout:[client] 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: rgw cache enabled = true 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable ops log = true 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable usage log = true 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: debug rbd = 20 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: debug rbd_mirror = 15 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: log to stderr = False 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: rbd default features = 125 2024-02-10T01:00:49.847 INFO:teuthology.orchestra.run.smithi008.stdout: rbd mirror image policy type = simple 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.a] 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.0] 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.1] 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.2] 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.3] 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.848 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.8 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = c022c2cc-868d-45e0-bbf6-c3f8e32882f5 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: bluefs allocator = hybrid 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore allocator = hybrid 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1/20 2024-02-10T01:00:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1/20 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 4/10 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-02-10T01:00:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-02-10T01:00:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-02-10T01:00:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:00:49.858 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd = 20 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd_mirror = 15 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: log to stderr = False 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: rbd default features = 125 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: rbd mirror image policy type = simple 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-02-10T01:00:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:00:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:00:49.861 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-02-10T01:00:49.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-02-10T01:00:49.916 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-10T01:00:49.917 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:49.917 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-02-10T01:00:49.931 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:49.931 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-02-10T01:00:49.986 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T01:00:49.986 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:49.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/cluster1.keyring 2024-02-10T01:00:49.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-02-10T01:00:50.064 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:00:50.064 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:50.119 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T01:00:50.119 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:00:50.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/cluster1.keyring 2024-02-10T01:00:50.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-02-10T01:00:50.161 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:00:50.161 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:50.216 INFO:tasks.ceph:Setting up mon nodes... 2024-02-10T01:00:50.216 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-10T01:00:50.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-02-10T01:00:50.270 INFO:teuthology.orchestra.run.smithi008.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-02-10T01:00:50.272 INFO:tasks.ceph:Setting up mds nodes... 2024-02-10T01:00:50.272 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-10T01:00:50.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-02-10T01:00:50.313 INFO:teuthology.orchestra.run.smithi008.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-02-10T01:00:50.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-02-10T01:00:50.379 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-02-10T01:00:50.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-02-10T01:00:50.436 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-02-10T01:00:50.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-02-10T01:00:50.534 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-02-10T01:00:50.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-02-10T01:00:50.630 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-02-10T01:00:50.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-02-10T01:00:50.728 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-02-10T01:00:50.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-02-10T01:00:50.824 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-02-10T01:00:50.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-02-10T01:00:50.921 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-02-10T01:00:50.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-02-10T01:00:51.018 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-02-10T01:00:51.033 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-10T01:00:51.033 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi008.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-10T01:00:51.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-02-10T01:00:51.065 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:00:51.065 INFO:tasks.ceph:role: cluster1.osd.0 2024-02-10T01:00:51.065 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T01:00:51.065 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-10T01:00:51.134 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-10T01:00:51.134 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-10T01:00:51.134 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-10T01:00:51.134 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-10T01:00:51.135 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-10T01:00:51.135 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-02-10T01:00:51.135 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-10T01:00:51.135 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-10T01:00:51.135 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-10T01:00:51.135 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-10T01:00:51.167 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-02-10T01:00:51.168 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-02-10T01:00:51.168 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-02-10T01:00:51.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-02-10T01:00:51.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-02-10T01:00:51.386 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:00:51.387 INFO:tasks.ceph:role: cluster1.osd.1 2024-02-10T01:00:51.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T01:00:51.387 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-10T01:00:51.453 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-10T01:00:51.453 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-10T01:00:51.453 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-10T01:00:51.453 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-10T01:00:51.454 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-10T01:00:51.454 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-02-10T01:00:51.454 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-10T01:00:51.454 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-10T01:00:51.454 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-10T01:00:51.454 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-10T01:00:51.486 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-02-10T01:00:51.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-02-10T01:00:51.489 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-02-10T01:00:51.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-02-10T01:00:51.630 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-02-10T01:00:51.698 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:00:51.698 INFO:tasks.ceph:role: cluster1.osd.2 2024-02-10T01:00:51.698 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T01:00:51.698 DEBUG:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-10T01:00:51.767 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-10T01:00:51.768 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-10T01:00:51.768 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-10T01:00:51.799 INFO:teuthology.orchestra.run.smithi008.stdout:Discarding blocks...Done. 2024-02-10T01:00:51.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2024-02-10T01:00:51.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-02-10T01:00:51.877 DEBUG:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-02-10T01:00:51.947 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:52.065 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:52.062+0000 7f6705c11640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-02-10T01:00:52.065 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:52.063+0000 7f6705c11640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-02-10T01:00:52.065 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:52.063+0000 7f6705c11640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-02-10T01:00:52.065 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:52.063+0000 7f6705c11640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-02-10T01:00:52.065 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:52.063+0000 7f6705c11640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-02-10T01:00:52.065 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:52.063+0000 7f6705c11640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-02-10T01:00:54.911 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-02-10T01:00:54.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:55.034 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:55.032+0000 7f51d5106640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-02-10T01:00:55.035 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:55.032+0000 7f51d5106640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-02-10T01:00:55.035 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:55.032+0000 7f51d5106640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-02-10T01:00:55.035 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:55.032+0000 7f51d5106640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-02-10T01:00:55.035 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:55.032+0000 7f51d5106640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-02-10T01:00:55.035 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:55.032+0000 7f51d5106640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-02-10T01:00:57.897 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-02-10T01:00:57.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:00:58.016 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:58.013+0000 7fe6719fa640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-02-10T01:00:58.016 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:58.013+0000 7fe6719fa640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-02-10T01:00:58.016 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:58.013+0000 7fe6719fa640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-02-10T01:00:58.016 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:58.013+0000 7fe6719fa640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-02-10T01:00:58.016 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:58.013+0000 7fe6719fa640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-02-10T01:00:58.016 INFO:teuthology.orchestra.run.smithi008.stderr:2024-02-10T01:00:58.014+0000 7fe6719fa640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-02-10T01:01:00.861 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-02-10T01:01:00.892 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-10T01:01:00.892 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:00.892 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-02-10T01:01:00.963 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:00.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-02-10T01:01:01.032 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:01.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-02-10T01:01:01.102 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:01.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-02-10T01:01:01.172 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:01.172 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-02-10T01:01:01.227 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.228 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-02-10T01:01:01.246 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.246 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-02-10T01:01:01.303 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.304 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-02-10T01:01:01.361 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.362 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-02-10T01:01:01.418 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.418 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-02-10T01:01:01.476 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.476 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-02-10T01:01:01.533 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.534 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-02-10T01:01:01.591 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:01.592 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-02-10T01:01:01.649 INFO:tasks.ceph:Adding keys to all mons... 2024-02-10T01:01:01.649 DEBUG:teuthology.orchestra.run.smithi008:> sudo tee -a /etc/ceph/cluster1.keyring 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr.x] 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZlwrEZEBAAx7V6CTSOeKtDLmNWniVDzw== 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.0] 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDEysZlDMLbAxAAkdMErkseRsx3zFyMvOIRWw== 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.1] 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDHysZlMnIJAhAAFfM/Jnz7I+I4UvFm7D+7sg== 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.2] 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDKysZl4WTtABAAzZZ0X9DVp0HgMUm1SNwRlg== 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZlrOazEhAAVLXznjvv+I6X4/JuFmVUHw== 2024-02-10T01:01:01.679 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZliFGaFhAAjspKwHkrVIf/vGGGYikX0A== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.0] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZlom4EGhAAdBeqgw3ypT2+2XHI1ZEFMA== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.1] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZl3hXTHxAAl05Zb3Tf9ovUjUUAeAdbKA== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.2] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZlbcOVJRAA6iCLpY7F75g5EH7eYCPpkQ== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.3] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZl0HVkKxAAMRQECJtGofnYeQi5uKeqVw== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.4] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZl1zkqMRAASPvwhMjksvI46GPZt/HxpQ== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.5] 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDCysZl7pntNhAA+Rq7bY4etMp/KRrh+GRD/Q== 2024-02-10T01:01:01.680 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.6] 2024-02-10T01:01:01.681 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDDysZlBqUQARAAwJfUX2NDfhS2J6zu81l0yg== 2024-02-10T01:01:01.681 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-10T01:01:01.780 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-10T01:01:01.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-10T01:01:01.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-10T01:01:01.939 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:01.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.047 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.101 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.155 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.210 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:02.426 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-10T01:01:02.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-02-10T01:01:02.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-02-10T01:01:02.548 DEBUG:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-02-10T01:01:02.575 DEBUG:teuthology.orchestra.run.smithi008:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-02-10T01:01:02.629 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-02-10T01:01:02.629 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-02-10T01:01:02.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-02-10T01:01:02.671 INFO:tasks.ceph.cluster1.mon.a:Started 2024-02-10T01:01:02.671 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-02-10T01:01:02.672 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-02-10T01:01:02.672 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-02-10T01:01:02.674 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-02-10T01:01:02.674 INFO:tasks.ceph:Setting crush tunables to default 2024-02-10T01:01:02.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster cluster1 osd crush tunables default 2024-02-10T01:01:02.861 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2024-02-10T01:01:02.872 INFO:tasks.ceph:check_enable_crimson: False 2024-02-10T01:01:02.872 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-02-10T01:01:02.873 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:02.873 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-02-10T01:01:02.899 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:02.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-02-10T01:01:02.966 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:02.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-02-10T01:01:03.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster cluster1 osd new 31e114d3-a231-480e-b4f7-4d456fce2b58 0 2024-02-10T01:01:03.258 INFO:teuthology.orchestra.run.smithi008.stdout:0 2024-02-10T01:01:03.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster cluster1 osd new 2d584223-c033-431a-94b2-7d9dd15ed0d9 1 2024-02-10T01:01:03.425 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:03.423+0000 7fa4a1811040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T01:01:03.456 INFO:teuthology.orchestra.run.smithi008.stdout:1 2024-02-10T01:01:03.464 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster cluster1 osd new 35ad0fe2-25a0-453f-8782-1593acce1103 2 2024-02-10T01:01:03.482 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:03.481+0000 7fa4a1811040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T01:01:03.648 INFO:teuthology.orchestra.run.smithi008.stdout:2 2024-02-10T01:01:03.660 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-02-10T01:01:03.660 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-02-10T01:01:03.703 INFO:tasks.ceph.cluster1.osd.0:Started 2024-02-10T01:01:03.704 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-02-10T01:01:03.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-02-10T01:01:03.706 INFO:tasks.ceph.cluster1.osd.1:Started 2024-02-10T01:01:03.706 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-02-10T01:01:03.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-02-10T01:01:03.708 INFO:tasks.ceph.cluster1.osd.2:Started 2024-02-10T01:01:03.708 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-02-10T01:01:03.904 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:03.904 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":5,"fsid":"c022c2cc-868d-45e0-bbf6-c3f8e32882f5","created":"2024-02-10T01:01:02.779834+0000","modified":"2024-02-10T01:01:03.648043+0000","last_up_change":"0.000000","last_in_change":"2024-02-10T01:01:03.648043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"31e114d3-a231-480e-b4f7-4d456fce2b58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2d584223-c033-431a-94b2-7d9dd15ed0d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35ad0fe2-25a0-453f-8782-1593acce1103","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:03.915 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-02-10T01:01:03.916 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-10T01:01:04.057 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:01:04.055+0000 7fc9b8e68640 -1 Falling back to public interface 2024-02-10T01:01:04.060 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.058+0000 7fa4a1811040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T01:01:04.116 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.114+0000 7fa4a1811040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T01:01:04.163 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.161+0000 7fa4a1811040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T01:01:04.218 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-02-10T01:01:04.218 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.216+0000 7fa4a1811040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T01:01:04.265 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.262+0000 7fa4a1811040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T01:01:04.306 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:01:04.304+0000 7feaaf437640 -1 Falling back to public interface 2024-02-10T01:01:04.323 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:01:04.321+0000 7f78c1666640 -1 Falling back to public interface 2024-02-10T01:01:04.387 INFO:teuthology.misc.health.smithi008.stdout: 2024-02-10T01:01:04.387 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":5,"fsid":"c022c2cc-868d-45e0-bbf6-c3f8e32882f5","created":"2024-02-10T01:01:02.779834+0000","modified":"2024-02-10T01:01:03.648043+0000","last_up_change":"0.000000","last_in_change":"2024-02-10T01:01:03.648043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"31e114d3-a231-480e-b4f7-4d456fce2b58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2d584223-c033-431a-94b2-7d9dd15ed0d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35ad0fe2-25a0-453f-8782-1593acce1103","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:04.390 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.387+0000 7fa4a1811040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T01:01:04.396 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-10T01:01:04.463 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.461+0000 7fa4a1811040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T01:01:04.511 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.508+0000 7fa4a1811040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T01:01:04.565 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.563+0000 7fa4a1811040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T01:01:04.613 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:04.612+0000 7fa4a1811040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T01:01:05.009 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.007+0000 7fa4a1811040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T01:01:05.149 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.147+0000 7fa4a1811040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T01:01:05.299 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.296+0000 7fa4a1811040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T01:01:05.346 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.344+0000 7fa4a1811040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T01:01:05.462 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.460+0000 7fa4a1811040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T01:01:05.560 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.558+0000 7fa4a1811040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T01:01:05.721 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.719+0000 7fa4a1811040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T01:01:05.765 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.763+0000 7fa4a1811040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T01:01:05.925 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:05.923+0000 7fa4a1811040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T01:01:06.764 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:06.762+0000 7fa4a1811040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T01:01:07.061 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:07.059+0000 7fa4a1811040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T01:01:07.127 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:07.125+0000 7fa4a1811040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T01:01:07.250 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:07.248+0000 7fa4a1811040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T01:01:07.535 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:01:07.533+0000 7fc9b8e68640 -1 osd.0 0 log_to_monitors true 2024-02-10T01:01:07.794 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:01:07.792+0000 7feaaf437640 -1 osd.2 0 log_to_monitors true 2024-02-10T01:01:07.843 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:01:07.841+0000 7f78c1666640 -1 osd.1 0 log_to_monitors true 2024-02-10T01:01:08.319 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:01:08.317+0000 7fa442055640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-10T01:01:10.700 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-02-10T01:01:10.753 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:01:10.752+0000 7f78bd5fc640 -1 osd.1 0 waiting for initial osdmap 2024-02-10T01:01:10.765 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:01:10.764+0000 7feaab3cd640 -1 osd.2 0 waiting for initial osdmap 2024-02-10T01:01:10.770 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:01:10.769+0000 7f78b541c640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T01:01:10.780 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:01:10.779+0000 7feaa31ed640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T01:01:10.785 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:01:10.784+0000 7fc9b5612640 -1 osd.0 0 waiting for initial osdmap 2024-02-10T01:01:10.797 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:01:10.796+0000 7fc9acc1e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T01:01:11.011 INFO:teuthology.misc.health.smithi008.stdout: 2024-02-10T01:01:11.011 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":7,"fsid":"c022c2cc-868d-45e0-bbf6-c3f8e32882f5","created":"2024-02-10T01:01:02.779834+0000","modified":"2024-02-10T01:01:09.504073+0000","last_up_change":"0.000000","last_in_change":"2024-02-10T01:01:03.648043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"31e114d3-a231-480e-b4f7-4d456fce2b58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2d584223-c033-431a-94b2-7d9dd15ed0d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35ad0fe2-25a0-453f-8782-1593acce1103","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:11.022 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-10T01:01:17.326 DEBUG:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-02-10T01:01:17.623 INFO:teuthology.misc.health.smithi008.stdout: 2024-02-10T01:01:17.623 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":12,"fsid":"c022c2cc-868d-45e0-bbf6-c3f8e32882f5","created":"2024-02-10T01:01:02.779834+0000","modified":"2024-02-10T01:01:15.773527+0000","last_up_change":"2024-02-10T01:01:11.766023+0000","last_in_change":"2024-02-10T01:01:03.648043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T01:01:13.508191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31e114d3-a231-480e-b4f7-4d456fce2b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6801","nonce":1194405554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6803","nonce":1194405554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6807","nonce":1194405554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6805","nonce":1194405554}]},"public_addr":"172.21.15.8:6801/1194405554","cluster_addr":"172.21.15.8:6803/1194405554","heartbeat_back_addr":"172.21.15.8:6807/1194405554","heartbeat_front_addr":"172.21.15.8:6805/1194405554","state":["exists","up"]},{"osd":1,"uuid":"2d584223-c033-431a-94b2-7d9dd15ed0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6817","nonce":1881173740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6819","nonce":1881173740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6823","nonce":1881173740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6821","nonce":1881173740}]},"public_addr":"172.21.15.8:6817/1881173740","cluster_addr":"172.21.15.8:6819/1881173740","heartbeat_back_addr":"172.21.15.8:6823/1881173740","heartbeat_front_addr":"172.21.15.8:6821/1881173740","state":["exists","up"]},{"osd":2,"uuid":"35ad0fe2-25a0-453f-8782-1593acce1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6809","nonce":2097129948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6811","nonce":2097129948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6815","nonce":2097129948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6813","nonce":2097129948}]},"public_addr":"172.21.15.8:6809/2097129948","cluster_addr":"172.21.15.8:6811/2097129948","heartbeat_back_addr":"172.21.15.8:6815/2097129948","heartbeat_front_addr":"172.21.15.8:6813/2097129948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.498858+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.830069+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.789605+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:17.634 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-02-10T01:01:17.634 INFO:tasks.ceph:Creating RBD pool 2024-02-10T01:01:17.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-02-10T01:01:18.782 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'rbd' created 2024-02-10T01:01:18.800 DEBUG:teuthology.orchestra.run.smithi008:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-10T01:01:19.785 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-10T01:01:19.805 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-02-10T01:01:19.805 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-02-10T01:01:19.805 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-02-10T01:01:19.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-02-10T01:01:19.806 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-10T01:01:20.152 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:20.163 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":2099245888},{"type":"v1","addr":"172.21.15.8:6825","nonce":2099245888}]},"active_addr":"172.21.15.8:6825/2099245888","active_change":"2024-02-10T01:01:07.262634+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":2724822778}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":3526275059}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":1492334168}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":2339595566}]}]} 2024-02-10T01:01:20.168 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-02-10T01:01:20.168 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-02-10T01:01:20.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-02-10T01:01:20.462 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:20.463 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":14,"fsid":"c022c2cc-868d-45e0-bbf6-c3f8e32882f5","created":"2024-02-10T01:01:02.779834+0000","modified":"2024-02-10T01:01:19.782437+0000","last_up_change":"2024-02-10T01:01:11.766023+0000","last_in_change":"2024-02-10T01:01:03.648043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T01:01:13.508191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-10T01:01:17.948399+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31e114d3-a231-480e-b4f7-4d456fce2b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6801","nonce":1194405554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6803","nonce":1194405554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6807","nonce":1194405554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6805","nonce":1194405554}]},"public_addr":"172.21.15.8:6801/1194405554","cluster_addr":"172.21.15.8:6803/1194405554","heartbeat_back_addr":"172.21.15.8:6807/1194405554","heartbeat_front_addr":"172.21.15.8:6805/1194405554","state":["exists","up"]},{"osd":1,"uuid":"2d584223-c033-431a-94b2-7d9dd15ed0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6817","nonce":1881173740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6819","nonce":1881173740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6823","nonce":1881173740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6821","nonce":1881173740}]},"public_addr":"172.21.15.8:6817/1881173740","cluster_addr":"172.21.15.8:6819/1881173740","heartbeat_back_addr":"172.21.15.8:6823/1881173740","heartbeat_front_addr":"172.21.15.8:6821/1881173740","state":["exists","up"]},{"osd":2,"uuid":"35ad0fe2-25a0-453f-8782-1593acce1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6809","nonce":2097129948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6811","nonce":2097129948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6815","nonce":2097129948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6813","nonce":2097129948}]},"public_addr":"172.21.15.8:6809/2097129948","cluster_addr":"172.21.15.8:6811/2097129948","heartbeat_back_addr":"172.21.15.8:6815/2097129948","heartbeat_front_addr":"172.21.15.8:6813/2097129948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.498858+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.830069+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.789605+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:20.475 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-02-10T01:01:20.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-02-10T01:01:20.766 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:20.766 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":14,"fsid":"c022c2cc-868d-45e0-bbf6-c3f8e32882f5","created":"2024-02-10T01:01:02.779834+0000","modified":"2024-02-10T01:01:19.782437+0000","last_up_change":"2024-02-10T01:01:11.766023+0000","last_in_change":"2024-02-10T01:01:03.648043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T01:01:13.508191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-10T01:01:17.948399+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31e114d3-a231-480e-b4f7-4d456fce2b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6801","nonce":1194405554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6803","nonce":1194405554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6807","nonce":1194405554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1194405554},{"type":"v1","addr":"172.21.15.8:6805","nonce":1194405554}]},"public_addr":"172.21.15.8:6801/1194405554","cluster_addr":"172.21.15.8:6803/1194405554","heartbeat_back_addr":"172.21.15.8:6807/1194405554","heartbeat_front_addr":"172.21.15.8:6805/1194405554","state":["exists","up"]},{"osd":1,"uuid":"2d584223-c033-431a-94b2-7d9dd15ed0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6817","nonce":1881173740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6819","nonce":1881173740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6823","nonce":1881173740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":1881173740},{"type":"v1","addr":"172.21.15.8:6821","nonce":1881173740}]},"public_addr":"172.21.15.8:6817/1881173740","cluster_addr":"172.21.15.8:6819/1881173740","heartbeat_back_addr":"172.21.15.8:6823/1881173740","heartbeat_front_addr":"172.21.15.8:6821/1881173740","state":["exists","up"]},{"osd":2,"uuid":"35ad0fe2-25a0-453f-8782-1593acce1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6809","nonce":2097129948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6811","nonce":2097129948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6815","nonce":2097129948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2097129948},{"type":"v1","addr":"172.21.15.8:6813","nonce":2097129948}]},"public_addr":"172.21.15.8:6809/2097129948","cluster_addr":"172.21.15.8:6811/2097129948","heartbeat_back_addr":"172.21.15.8:6815/2097129948","heartbeat_front_addr":"172.21.15.8:6813/2097129948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.498858+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.830069+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:08.789605+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:20.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-02-10T01:01:20.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-02-10T01:01:20.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-02-10T01:01:20.921 INFO:teuthology.orchestra.run.smithi008.stdout:34359738372 2024-02-10T01:01:20.921 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-02-10T01:01:20.922 INFO:teuthology.orchestra.run.smithi008.stdout:34359738371 2024-02-10T01:01:20.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-02-10T01:01:20.923 INFO:teuthology.orchestra.run.smithi008.stdout:34359738372 2024-02-10T01:01:20.924 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-02-10T01:01:21.261 INFO:teuthology.orchestra.run.smithi008.stdout:34359738370 2024-02-10T01:01:21.266 INFO:teuthology.orchestra.run.smithi008.stdout:34359738371 2024-02-10T01:01:21.270 INFO:teuthology.orchestra.run.smithi008.stdout:34359738371 2024-02-10T01:01:21.275 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.2 2024-02-10T01:01:21.280 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2024-02-10T01:01:21.283 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-02-10T01:01:22.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-02-10T01:01:22.281 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-02-10T01:01:22.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-02-10T01:01:22.600 INFO:teuthology.orchestra.run.smithi008.stdout:34359738372 2024-02-10T01:01:22.603 INFO:teuthology.orchestra.run.smithi008.stdout:34359738371 2024-02-10T01:01:22.614 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2024-02-10T01:01:22.614 DEBUG:teuthology.parallel:result is None 2024-02-10T01:01:22.616 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2024-02-10T01:01:22.616 DEBUG:teuthology.parallel:result is None 2024-02-10T01:01:22.640 INFO:teuthology.orchestra.run.smithi008.stdout:34359738372 2024-02-10T01:01:22.653 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-02-10T01:01:22.653 DEBUG:teuthology.parallel:result is None 2024-02-10T01:01:22.653 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-02-10T01:01:22.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-02-10T01:01:22.980 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:22.981 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T01:01:22.992 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-02-10T01:01:21.268203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95064,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020456,"statfs":{"total":289910292480,"available":289812946944,"internally_reserved":0,"allocated":1556480,"data_stored":1282004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.485696"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789193+0000","last_change":"2024-02-10T01:01:19.792868+0000","last_active":"2024-02-10T01:01:20.789193+0000","last_peered":"2024-02-10T01:01:20.789193+0000","last_clean":"2024-02-10T01:01:20.789193+0000","last_became_active":"2024-02-10T01:01:19.792604+0000","last_became_peered":"2024-02-10T01:01:19.792604+0000","last_unstale":"2024-02-10T01:01:20.789193+0000","last_undegraded":"2024-02-10T01:01:20.789193+0000","last_fullsized":"2024-02-10T01:01:20.789193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:10:12.255846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789002+0000","last_change":"2024-02-10T01:01:19.794738+0000","last_active":"2024-02-10T01:01:20.789002+0000","last_peered":"2024-02-10T01:01:20.789002+0000","last_clean":"2024-02-10T01:01:20.789002+0000","last_became_active":"2024-02-10T01:01:19.794423+0000","last_became_peered":"2024-02-10T01:01:19.794423+0000","last_unstale":"2024-02-10T01:01:20.789002+0000","last_undegraded":"2024-02-10T01:01:20.789002+0000","last_fullsized":"2024-02-10T01:01:20.789002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:06:26.297737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789091+0000","last_change":"2024-02-10T01:01:19.794668+0000","last_active":"2024-02-10T01:01:20.789091+0000","last_peered":"2024-02-10T01:01:20.789091+0000","last_clean":"2024-02-10T01:01:20.789091+0000","last_became_active":"2024-02-10T01:01:19.794364+0000","last_became_peered":"2024-02-10T01:01:19.794364+0000","last_unstale":"2024-02-10T01:01:20.789091+0000","last_undegraded":"2024-02-10T01:01:20.789091+0000","last_fullsized":"2024-02-10T01:01:20.789091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:56:19.021096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.788996+0000","last_change":"2024-02-10T01:01:19.794602+0000","last_active":"2024-02-10T01:01:20.788996+0000","last_peered":"2024-02-10T01:01:20.788996+0000","last_clean":"2024-02-10T01:01:20.788996+0000","last_became_active":"2024-02-10T01:01:19.794320+0000","last_became_peered":"2024-02-10T01:01:19.794320+0000","last_unstale":"2024-02-10T01:01:20.788996+0000","last_undegraded":"2024-02-10T01:01:20.788996+0000","last_fullsized":"2024-02-10T01:01:20.788996+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:35:26.572065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:19.793897+0000","last_change":"2024-02-10T01:01:19.792574+0000","last_active":"2024-02-10T01:01:19.793897+0000","last_peered":"2024-02-10T01:01:19.793897+0000","last_clean":"2024-02-10T01:01:19.793897+0000","last_became_active":"2024-02-10T01:01:19.792247+0000","last_became_peered":"2024-02-10T01:01:19.792247+0000","last_unstale":"2024-02-10T01:01:19.793897+0000","last_undegraded":"2024-02-10T01:01:19.793897+0000","last_fullsized":"2024-02-10T01:01:19.793897+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:03:41.795619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:19.793987+0000","last_change":"2024-02-10T01:01:19.792589+0000","last_active":"2024-02-10T01:01:19.793987+0000","last_peered":"2024-02-10T01:01:19.793987+0000","last_clean":"2024-02-10T01:01:19.793987+0000","last_became_active":"2024-02-10T01:01:19.792245+0000","last_became_peered":"2024-02-10T01:01:19.792245+0000","last_unstale":"2024-02-10T01:01:19.793987+0000","last_undegraded":"2024-02-10T01:01:19.793987+0000","last_fullsized":"2024-02-10T01:01:19.793987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:43:00.942382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:19.793867+0000","last_change":"2024-02-10T01:01:19.792563+0000","last_active":"2024-02-10T01:01:19.793867+0000","last_peered":"2024-02-10T01:01:19.793867+0000","last_clean":"2024-02-10T01:01:19.793867+0000","last_became_active":"2024-02-10T01:01:19.792258+0000","last_became_peered":"2024-02-10T01:01:19.792258+0000","last_unstale":"2024-02-10T01:01:19.793867+0000","last_undegraded":"2024-02-10T01:01:19.793867+0000","last_fullsized":"2024-02-10T01:01:19.793867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:45:18.455304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789076+0000","last_change":"2024-02-10T01:01:19.795279+0000","last_active":"2024-02-10T01:01:20.789076+0000","last_peered":"2024-02-10T01:01:20.789076+0000","last_clean":"2024-02-10T01:01:20.789076+0000","last_became_active":"2024-02-10T01:01:19.793498+0000","last_became_peered":"2024-02-10T01:01:19.793498+0000","last_unstale":"2024-02-10T01:01:20.789076+0000","last_undegraded":"2024-02-10T01:01:20.789076+0000","last_fullsized":"2024-02-10T01:01:20.789076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:45:22.585192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789074+0000","last_change":"2024-02-10T01:01:14.776255+0000","last_active":"2024-02-10T01:01:20.789074+0000","last_peered":"2024-02-10T01:01:20.789074+0000","last_clean":"2024-02-10T01:01:20.789074+0000","last_became_active":"2024-02-10T01:01:14.775911+0000","last_became_peered":"2024-02-10T01:01:14.775911+0000","last_unstale":"2024-02-10T01:01:20.789074+0000","last_undegraded":"2024-02-10T01:01:20.789074+0000","last_fullsized":"2024-02-10T01:01:20.789074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:13.767753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:13.767753+0000","last_clean_scrub_stamp":"2024-02-10T01:01:13.767753+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:12:08.219786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T01:01:22.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-02-10T01:01:23.283 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:23.284 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T01:01:23.295 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-02-10T01:01:23.268551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95076,"kb_used_data":1528,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020444,"statfs":{"total":289910292480,"available":289812934656,"internally_reserved":0,"allocated":1564672,"data_stored":1285482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.486045"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789193+0000","last_change":"2024-02-10T01:01:19.792868+0000","last_active":"2024-02-10T01:01:20.789193+0000","last_peered":"2024-02-10T01:01:20.789193+0000","last_clean":"2024-02-10T01:01:20.789193+0000","last_became_active":"2024-02-10T01:01:19.792604+0000","last_became_peered":"2024-02-10T01:01:19.792604+0000","last_unstale":"2024-02-10T01:01:20.789193+0000","last_undegraded":"2024-02-10T01:01:20.789193+0000","last_fullsized":"2024-02-10T01:01:20.789193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:10:12.255846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789002+0000","last_change":"2024-02-10T01:01:19.794738+0000","last_active":"2024-02-10T01:01:20.789002+0000","last_peered":"2024-02-10T01:01:20.789002+0000","last_clean":"2024-02-10T01:01:20.789002+0000","last_became_active":"2024-02-10T01:01:19.794423+0000","last_became_peered":"2024-02-10T01:01:19.794423+0000","last_unstale":"2024-02-10T01:01:20.789002+0000","last_undegraded":"2024-02-10T01:01:20.789002+0000","last_fullsized":"2024-02-10T01:01:20.789002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:06:26.297737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789091+0000","last_change":"2024-02-10T01:01:19.794668+0000","last_active":"2024-02-10T01:01:20.789091+0000","last_peered":"2024-02-10T01:01:20.789091+0000","last_clean":"2024-02-10T01:01:20.789091+0000","last_became_active":"2024-02-10T01:01:19.794364+0000","last_became_peered":"2024-02-10T01:01:19.794364+0000","last_unstale":"2024-02-10T01:01:20.789091+0000","last_undegraded":"2024-02-10T01:01:20.789091+0000","last_fullsized":"2024-02-10T01:01:20.789091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:56:19.021096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.788996+0000","last_change":"2024-02-10T01:01:19.794602+0000","last_active":"2024-02-10T01:01:20.788996+0000","last_peered":"2024-02-10T01:01:20.788996+0000","last_clean":"2024-02-10T01:01:20.788996+0000","last_became_active":"2024-02-10T01:01:19.794320+0000","last_became_peered":"2024-02-10T01:01:19.794320+0000","last_unstale":"2024-02-10T01:01:20.788996+0000","last_undegraded":"2024-02-10T01:01:20.788996+0000","last_fullsized":"2024-02-10T01:01:20.788996+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:35:26.572065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:22.039088+0000","last_change":"2024-02-10T01:01:19.792574+0000","last_active":"2024-02-10T01:01:22.039088+0000","last_peered":"2024-02-10T01:01:22.039088+0000","last_clean":"2024-02-10T01:01:22.039088+0000","last_became_active":"2024-02-10T01:01:19.792247+0000","last_became_peered":"2024-02-10T01:01:19.792247+0000","last_unstale":"2024-02-10T01:01:22.039088+0000","last_undegraded":"2024-02-10T01:01:22.039088+0000","last_fullsized":"2024-02-10T01:01:22.039088+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:47:13.001211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:19.793987+0000","last_change":"2024-02-10T01:01:19.792589+0000","last_active":"2024-02-10T01:01:19.793987+0000","last_peered":"2024-02-10T01:01:19.793987+0000","last_clean":"2024-02-10T01:01:19.793987+0000","last_became_active":"2024-02-10T01:01:19.792245+0000","last_became_peered":"2024-02-10T01:01:19.792245+0000","last_unstale":"2024-02-10T01:01:19.793987+0000","last_undegraded":"2024-02-10T01:01:19.793987+0000","last_fullsized":"2024-02-10T01:01:19.793987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:43:00.942382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:19.793867+0000","last_change":"2024-02-10T01:01:19.792563+0000","last_active":"2024-02-10T01:01:19.793867+0000","last_peered":"2024-02-10T01:01:19.793867+0000","last_clean":"2024-02-10T01:01:19.793867+0000","last_became_active":"2024-02-10T01:01:19.792258+0000","last_became_peered":"2024-02-10T01:01:19.792258+0000","last_unstale":"2024-02-10T01:01:19.793867+0000","last_undegraded":"2024-02-10T01:01:19.793867+0000","last_fullsized":"2024-02-10T01:01:19.793867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:45:18.455304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789076+0000","last_change":"2024-02-10T01:01:19.795279+0000","last_active":"2024-02-10T01:01:20.789076+0000","last_peered":"2024-02-10T01:01:20.789076+0000","last_clean":"2024-02-10T01:01:20.789076+0000","last_became_active":"2024-02-10T01:01:19.793498+0000","last_became_peered":"2024-02-10T01:01:19.793498+0000","last_unstale":"2024-02-10T01:01:20.789076+0000","last_undegraded":"2024-02-10T01:01:20.789076+0000","last_fullsized":"2024-02-10T01:01:20.789076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:18.779079+0000","last_clean_scrub_stamp":"2024-02-10T01:01:18.779079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:45:22.585192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-02-10T01:01:20.789074+0000","last_change":"2024-02-10T01:01:14.776255+0000","last_active":"2024-02-10T01:01:20.789074+0000","last_peered":"2024-02-10T01:01:20.789074+0000","last_clean":"2024-02-10T01:01:20.789074+0000","last_became_active":"2024-02-10T01:01:14.775911+0000","last_became_peered":"2024-02-10T01:01:14.775911+0000","last_unstale":"2024-02-10T01:01:20.789074+0000","last_undegraded":"2024-02-10T01:01:20.789074+0000","last_fullsized":"2024-02-10T01:01:20.789074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:13.767753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:13.767753+0000","last_clean_scrub_stamp":"2024-02-10T01:01:13.767753+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:12:08.219786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":724992,"data_stored":627602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T01:01:23.296 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-02-10T01:01:23.296 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-02-10T01:01:23.296 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-02-10T01:01:23.296 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-02-10T01:01:23.623 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:23.623 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T01:01:23.635 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-02-10T01:01:23.635 INFO:teuthology.run_tasks:Running task ceph... 2024-02-10T01:01:23.646 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-02-10T01:01:23.646 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-10T01:01:23.646 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7554696', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5679, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'targets': {'smithi008.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlSAKYrhH4t6lqydJwPc7dakwrUcEIWsSCqIlLXUX8KcGMzn9wmRKv3395yFBqkRi00GYbssmuqVssZshiM65E=', 'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3fOJ0AL93zBAlc9KUWLcYTG+/ZN22p+OeOUdu5BJmCIqUtDwSyLdMZQynzTw6sRiugO8+IUpLWGgYqOxmkkzs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd9fdb2209e15b39d9f061fd85399f352ce0f0894', 'timestamp': '2024-02-09_19:39:00', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-10T01:01:23.647 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-02-10T01:01:23.650 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-02-10T01:01:23.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/run/ceph 2024-02-10T01:01:23.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-02-10T01:01:23.740 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:23.740 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T01:01:23.797 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T01:01:23.797 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-02-10T01:01:23.853 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T01:01:23.853 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:01:23.853 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T01:01:23.853 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:01:23.853 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:01:23.854 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-02-10 01:00:23.706300540 +0000 2024-02-10T01:01:23.854 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-02-10 00:57:31.231167706 +0000 2024-02-10T01:01:23.854 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-02-10 00:57:31.231167706 +0000 2024-02-10T01:01:23.854 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-02-10 00:57:31.231167706 +0000 2024-02-10T01:01:23.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T01:01:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-02-10T01:01:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-02-10T01:01:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000108848 s, 4.7 MB/s 2024-02-10T01:01:23.923 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T01:01:23.981 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-02-10T01:01:24.038 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T01:01:24.038 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:01:24.038 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T01:01:24.038 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:01:24.039 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:01:24.039 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-02-10 01:00:23.707300531 +0000 2024-02-10T01:01:24.039 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-02-10 00:57:31.557164827 +0000 2024-02-10T01:01:24.039 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-02-10 00:57:31.557164827 +0000 2024-02-10T01:01:24.039 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-02-10 00:57:31.557164827 +0000 2024-02-10T01:01:24.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T01:01:24.110 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-02-10T01:01:24.110 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-02-10T01:01:24.110 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000107978 s, 4.7 MB/s 2024-02-10T01:01:24.111 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T01:01:24.169 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-02-10T01:01:24.224 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T01:01:24.224 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:01:24.224 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-10T01:01:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:01:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:01:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-02-10 01:00:23.707300531 +0000 2024-02-10T01:01:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-02-10 00:57:31.900161798 +0000 2024-02-10T01:01:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-02-10 00:57:31.900161798 +0000 2024-02-10T01:01:24.225 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-02-10 00:57:31.900161798 +0000 2024-02-10T01:01:24.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T01:01:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-02-10T01:01:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-02-10T01:01:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 8.8751e-05 s, 5.8 MB/s 2024-02-10T01:01:24.292 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T01:01:24.350 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-02-10 01:00:23.708300523 +0000 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-02-10 00:57:32.218158991 +0000 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-02-10 00:57:32.218158991 +0000 2024-02-10T01:01:24.405 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-02-10 00:57:32.218158991 +0000 2024-02-10T01:01:24.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T01:01:24.471 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-02-10T01:01:24.471 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-02-10T01:01:24.471 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 8.2433e-05 s, 6.2 MB/s 2024-02-10T01:01:24.473 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T01:01:24.532 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:01:24.533 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-10T01:01:24.533 INFO:tasks.ceph:Generating config... 2024-02-10T01:01:24.534 INFO:tasks.ceph:[client] debug rbd = 20 2024-02-10T01:01:24.534 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-02-10T01:01:24.534 INFO:tasks.ceph:[client] log to stderr = False 2024-02-10T01:01:24.534 INFO:tasks.ceph:[client] rbd default features = 125 2024-02-10T01:01:24.534 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:24.535 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:24.535 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-02-10T01:01:24.535 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-02-10T01:01:24.535 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-02-10T01:01:24.535 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-10T01:01:24.535 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-10T01:01:24.535 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-10T01:01:24.535 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-10T01:01:24.536 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-10T01:01:24.536 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-10T01:01:24.537 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-10T01:01:24.537 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-10T01:01:24.537 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-02-10T01:01:24.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-02-10T01:01:24.616 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.keyring 2024-02-10T01:01:24.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-02-10T01:01:24.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-02-10T01:01:24.769 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.60')] 2024-02-10T01:01:24.770 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-02-10T01:01:24.770 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:24.770 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-10T01:01:24.824 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --print /home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid 180b9984-42b1-4d69-a091-ca59eeed2339 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:fsid 180b9984-42b1-4d69-a091-ca59eeed2339 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-02-10T01:01:24.902763+0000 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-02-10T01:01:24.902763+0000 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 17 (quincy) 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-02-10T01:01:24.903 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-02-10T01:01:24.904 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-10T01:01:24.959 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 180b9984-42b1-4d69-a091-ca59eeed2339... 2024-02-10T01:01:24.960 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-02-10T01:01:24.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-02-10T01:01:25.017 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-02-10T01:01:25.017 INFO:teuthology.orchestra.run.smithi008.stdout: chdir = "" 2024-02-10T01:01:25.017 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-10T01:01:25.017 INFO:teuthology.orchestra.run.smithi008.stdout: auth supported = cephx 2024-02-10T01:01:25.017 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.017 INFO:teuthology.orchestra.run.smithi008.stdout: filestore xattr use omap = true 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: mon clock drift allowed = 1.000 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: osd crush chooseleaf type = 0 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: auth debug = true 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on old message = true 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on bug = true 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: mon pg warn max object skew = 0 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: # disable pg_autoscaler by default for new pools 2024-02-10T01:01:25.018 INFO:teuthology.orchestra.run.smithi008.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default size = 2 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow primary affinity = true 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow pg remap = true 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on legacy crush tunables = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on crush straw calc version zero = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on no sortbitwise = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on osd down out interval zero = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on too few osds = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: mon_allow_pool_size_one = true 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.019 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: osd default data pool replay window = 5 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: mon allow pool delete = true 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: mon cluster log file level = debug 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: debug asserts on shutdown = true 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: mon health detail to clog = false 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: mon host = 172.21.15.60 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: mon client directed command retry = 5 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on pool no app = False 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: ms inject socket failures = 5000 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = 180b9984-42b1-4d69-a091-ca59eeed2339 2024-02-10T01:01:25.020 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout:[osd] 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd journal size = 100 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub load threshold = 5.0 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub max interval = 600 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd mclock profile = high_recovery_ops 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd recover clone overlap = true 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd recovery max chunk = 1048576 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug shutdown = true 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug op order = true 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug verify stray on activate = true 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug trim objects = true 2024-02-10T01:01:25.021 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: osd open classes on start = true 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug pg log writeout = true 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: osd deep scrub update digest min age = 30 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: osd map max advance = 10 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: journal zero on create = true 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: filestore ondisk finisher threads = 3 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: filestore apply finisher threads = 3 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: bdev debug aio = true 2024-02-10T01:01:25.022 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug misdirected ops = true 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: bluefs allocator = hybrid 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore allocator = hybrid 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore block size = 96636764160 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore fsck on mount = True 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluefs = 1/20 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluestore = 1/20 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: debug osd = 20 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: debug rocksdb = 4/10 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd backfillfull_ratio = 0.85 2024-02-10T01:01:25.023 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd full ratio = 0.9 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd nearfull ratio = 0.8 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: osd failsafe full ratio = 0.95 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: osd objectstore = bluestore 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr] 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: debug mgr = 20 2024-02-10T01:01:25.024 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: mgr/telemetry/nag = false 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout:[mon] 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: debug paxos = 20 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: mon data avail warn = 5 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: mon mgr mkfs grace = 240 2024-02-10T01:01:25.025 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd reporter subtree level = osd 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd prime pg temp = true 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: auth mon ticket ttl = 660 # 11m 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: auth service ticket ttl = 240 # 4m 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: # don't complain about insecure global_id in the test suite 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: # 1m isn't quite enough 2024-02-10T01:01:25.026 INFO:teuthology.orchestra.run.smithi008.stdout: mon_down_mkfs_grace = 2m 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_filestore_osds = false 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout:[client] 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: rgw cache enabled = true 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable ops log = true 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable usage log = true 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: debug rbd = 20 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: debug rbd_mirror = 15 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: log to stderr = False 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: rbd default features = 125 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout: rbd mirror image policy type = simple 2024-02-10T01:01:25.027 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.a] 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.0] 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.1] 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.2] 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout:[client.mirror.3] 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.028 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.049 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-02-10T01:01:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.60 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 180b9984-42b1-4d69-a091-ca59eeed2339 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-02-10T01:01:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-02-10T01:01:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: bluefs allocator = hybrid 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore allocator = hybrid 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1/20 2024-02-10T01:01:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1/20 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 4/10 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-02-10T01:01:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-02-10T01:01:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-02-10T01:01:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd = 20 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd_mirror = 15 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: log to stderr = False 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: rbd default features = 125 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: rbd mirror image policy type = simple 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-02-10T01:01:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-02-10T01:01:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-02-10T01:01:25.064 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-02-10T01:01:25.064 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-02-10T01:01:25.159 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-10T01:01:25.159 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:25.159 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-02-10T01:01:25.174 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:25.174 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-02-10T01:01:25.228 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T01:01:25.228 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:25.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/cluster2.keyring 2024-02-10T01:01:25.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-02-10T01:01:25.273 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:25.273 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:25.329 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T01:01:25.329 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:25.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/cluster2.keyring 2024-02-10T01:01:25.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-02-10T01:01:25.369 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:25.369 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:25.424 INFO:tasks.ceph:Setting up mon nodes... 2024-02-10T01:01:25.425 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-10T01:01:25.425 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-02-10T01:01:25.483 INFO:teuthology.orchestra.run.smithi008.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-02-10T01:01:25.485 INFO:tasks.ceph:Setting up mds nodes... 2024-02-10T01:01:25.485 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-10T01:01:25.486 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-02-10T01:01:25.529 INFO:teuthology.orchestra.run.smithi008.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-02-10T01:01:25.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-02-10T01:01:25.586 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-02-10T01:01:25.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-02-10T01:01:25.682 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-02-10T01:01:25.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-02-10T01:01:25.779 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-02-10T01:01:25.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-02-10T01:01:25.876 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-02-10T01:01:25.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-02-10T01:01:25.975 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-02-10T01:01:25.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-02-10T01:01:26.072 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-02-10T01:01:26.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-02-10T01:01:26.168 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-02-10T01:01:26.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-02-10T01:01:26.265 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-02-10T01:01:26.280 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-10T01:01:26.281 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi008.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-10T01:01:26.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-02-10T01:01:26.347 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:01:26.347 INFO:tasks.ceph:role: cluster2.osd.0 2024-02-10T01:01:26.347 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T01:01:26.347 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-10T01:01:26.413 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-10T01:01:26.413 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-10T01:01:26.413 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-10T01:01:26.413 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-10T01:01:26.413 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-10T01:01:26.414 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-02-10T01:01:26.414 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-10T01:01:26.414 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-10T01:01:26.414 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-10T01:01:26.415 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-10T01:01:26.445 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-02-10T01:01:26.447 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-02-10T01:01:26.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-02-10T01:01:26.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-02-10T01:01:26.593 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-02-10T01:01:26.659 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:01:26.660 INFO:tasks.ceph:role: cluster2.osd.1 2024-02-10T01:01:26.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T01:01:26.660 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-10T01:01:26.726 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-10T01:01:26.758 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-02-10T01:01:26.760 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-02-10T01:01:26.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-02-10T01:01:26.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-02-10T01:01:26.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-02-10T01:01:26.968 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-02-10T01:01:26.968 INFO:tasks.ceph:role: cluster2.osd.2 2024-02-10T01:01:26.968 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T01:01:26.968 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-10T01:01:27.034 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-10T01:01:27.066 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-02-10T01:01:27.068 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-02-10T01:01:27.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-02-10T01:01:27.143 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-02-10T01:01:27.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:27.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:27.315+0000 7fa3ed012640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-02-10T01:01:27.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:27.315+0000 7fa3ed012640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-02-10T01:01:27.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:27.315+0000 7fa3ed012640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-02-10T01:01:27.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:27.315+0000 7fa3ed012640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-02-10T01:01:27.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:27.315+0000 7fa3ed012640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-02-10T01:01:27.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:27.315+0000 7fa3ed012640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-02-10T01:01:30.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-02-10T01:01:30.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:30.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:30.288+0000 7f03ca048640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-02-10T01:01:30.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:30.289+0000 7f03ca048640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-02-10T01:01:30.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:30.289+0000 7f03ca048640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-02-10T01:01:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:30.289+0000 7f03ca048640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-02-10T01:01:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:30.289+0000 7f03ca048640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-02-10T01:01:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:30.289+0000 7f03ca048640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-02-10T01:01:33.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-02-10T01:01:33.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:33.254+0000 7f93db211640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-02-10T01:01:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:33.254+0000 7f93db211640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-02-10T01:01:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:33.254+0000 7f93db211640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-02-10T01:01:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:33.254+0000 7f93db211640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-02-10T01:01:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:33.254+0000 7f93db211640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-02-10T01:01:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-02-10T01:01:33.255+0000 7f93db211640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-02-10T01:01:36.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-02-10T01:01:36.132 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-10T01:01:36.132 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:36.132 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-02-10T01:01:36.164 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-02-10T01:01:36.203 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-02-10T01:01:36.274 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-02-10T01:01:36.343 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T01:01:36.343 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-02-10T01:01:36.361 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.361 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-02-10T01:01:36.400 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.400 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-02-10T01:01:36.456 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.456 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-02-10T01:01:36.512 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.513 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-02-10T01:01:36.569 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.569 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-02-10T01:01:36.626 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.626 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-02-10T01:01:36.683 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.683 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-02-10T01:01:36.739 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:36.739 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-02-10T01:01:36.796 INFO:tasks.ceph:Adding keys to all mons... 2024-02-10T01:01:36.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/cluster2.keyring 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZl+hTNHBAAJWeAnvgXWKUmhyPSm5McIw== 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDnysZl9UDaEhAA3/cUYnXt7bHMu8jJSuR6Qg== 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDqysZlNq9HERAAAWzKhXeid4j/B5IY/eoQDQ== 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDtysZlkjw3DxAAtc8b/abJQj/220pMp2FoHA== 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlDneJHxAA2KzyG1uxP8krOSKpi3YUzw== 2024-02-10T01:01:36.865 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlrQ/zIhAARkNjGy5uc78exGkHoaQ6lg== 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlxkmxKBAASQBHlH/YKgmfr6zNF3ue1A== 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlNgt4LhAAAMNWEsGpZBAv4nBuSIw8gw== 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlUHY/NBAA4c2z+aYdl5Wo1CwRksNZXA== 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlWuYaOhAABpG1AD0Zsog6h2T7oP6TAg== 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.4] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDmysZly79LBBAA3/2NLSNabkxZBhOnAKxp5A== 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.5] 2024-02-10T01:01:36.866 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDmysZlaoAPChAAi0C2QMqQE7Y+aB5vYjpftA== 2024-02-10T01:01:36.867 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.6] 2024-02-10T01:01:36.867 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDmysZl+JPWDxAA+3KBoBWgd40qmZPhsa5Dhg== 2024-02-10T01:01:36.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-10T01:01:36.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-10T01:01:37.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-10T01:01:37.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-10T01:01:37.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.284 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.339 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.558 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-10T01:01:37.612 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-10T01:01:37.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-02-10T01:01:37.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-02-10T01:01:37.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-02-10T01:01:37.761 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-02-10T01:01:37.816 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-02-10T01:01:37.817 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-02-10T01:01:37.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-02-10T01:01:37.859 INFO:tasks.ceph.cluster2.mon.a:Started 2024-02-10T01:01:37.860 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-02-10T01:01:37.860 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-02-10T01:01:37.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-02-10T01:01:37.862 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-02-10T01:01:37.863 INFO:tasks.ceph:Setting crush tunables to default 2024-02-10T01:01:37.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd crush tunables default 2024-02-10T01:01:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-02-10T01:01:38.065 INFO:tasks.ceph:check_enable_crimson: False 2024-02-10T01:01:38.065 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-02-10T01:01:38.065 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:38.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-02-10T01:01:38.093 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:38.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-02-10T01:01:38.158 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:01:38.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-02-10T01:01:38.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd new b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e 0 2024-02-10T01:01:38.343 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.342+0000 7f50fb243040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T01:01:38.391 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.390+0000 7f50fb243040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T01:01:38.446 INFO:teuthology.orchestra.run.smithi060.stdout:0 2024-02-10T01:01:38.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd new f191b760-bfd1-4937-a959-47327a85bab1 1 2024-02-10T01:01:38.639 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-02-10T01:01:38.650 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd new d643becc-6621-49bb-9fe8-ae343e7a6317 2 2024-02-10T01:01:38.789 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.788+0000 7f50fb243040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T01:01:38.832 INFO:teuthology.orchestra.run.smithi060.stdout:2 2024-02-10T01:01:38.836 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.835+0000 7f50fb243040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T01:01:38.843 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-02-10T01:01:38.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-02-10T01:01:38.845 INFO:tasks.ceph.cluster2.osd.0:Started 2024-02-10T01:01:38.845 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-02-10T01:01:38.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-02-10T01:01:38.847 INFO:tasks.ceph.cluster2.osd.1:Started 2024-02-10T01:01:38.847 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-02-10T01:01:38.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-02-10T01:01:38.849 INFO:tasks.ceph.cluster2.osd.2:Started 2024-02-10T01:01:38.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-02-10T01:01:38.883 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.880+0000 7f50fb243040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T01:01:38.934 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.933+0000 7f50fb243040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T01:01:38.977 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:38.977+0000 7f50fb243040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T01:01:39.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:39.024 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"fsid":"180b9984-42b1-4d69-a091-ca59eeed2339","created":"2024-02-10T01:01:37.970961+0000","modified":"2024-02-10T01:01:38.831239+0000","last_up_change":"0.000000","last_in_change":"2024-02-10T01:01:38.831239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f191b760-bfd1-4937-a959-47327a85bab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d643becc-6621-49bb-9fe8-ae343e7a6317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:39.034 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-02-10T01:01:39.035 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-10T01:01:39.095 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.094+0000 7f50fb243040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T01:01:39.150 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.149+0000 7f50fb243040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T01:01:39.193 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.192+0000 7f50fb243040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T01:01:39.244 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.243+0000 7f50fb243040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T01:01:39.288 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.287+0000 7f50fb243040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T01:01:39.337 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-02-10T01:01:39.443 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:01:39.442+0000 7febd0d9e640 -1 Falling back to public interface 2024-02-10T01:01:39.444 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:01:39.442+0000 7f1f56973640 -1 Falling back to public interface 2024-02-10T01:01:39.444 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:01:39.443+0000 7f1f55922640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6807/0: (98) Address already in use 2024-02-10T01:01:39.444 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:01:39.443+0000 7febcfd4d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6811/0: (98) Address already in use 2024-02-10T01:01:39.447 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:01:39.447+0000 7fe730a0f640 -1 Falling back to public interface 2024-02-10T01:01:39.507 INFO:teuthology.misc.health.smithi060.stdout: 2024-02-10T01:01:39.508 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":5,"fsid":"180b9984-42b1-4d69-a091-ca59eeed2339","created":"2024-02-10T01:01:37.970961+0000","modified":"2024-02-10T01:01:38.831239+0000","last_up_change":"0.000000","last_in_change":"2024-02-10T01:01:38.831239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f191b760-bfd1-4937-a959-47327a85bab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d643becc-6621-49bb-9fe8-ae343e7a6317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:39.517 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-10T01:01:39.619 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.618+0000 7f50fb243040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T01:01:39.755 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.754+0000 7f50fb243040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T01:01:39.852 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.851+0000 7f50fb243040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T01:01:39.896 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:39.895+0000 7f50fb243040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T01:01:40.010 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:40.009+0000 7f50fb243040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T01:01:40.105 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:40.104+0000 7f50fb243040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T01:01:40.263 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:40.262+0000 7f50fb243040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T01:01:40.306 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:40.305+0000 7f50fb243040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T01:01:40.430 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:40.429+0000 7f50fb243040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T01:01:41.055 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:41.054+0000 7f50fb243040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T01:01:41.329 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:41.328+0000 7f50fb243040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T01:01:41.381 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:41.380+0000 7f50fb243040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T01:01:41.500 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:41.499+0000 7f50fb243040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T01:01:42.508 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:01:42.508+0000 7f509c005640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-10T01:01:42.904 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:01:42.903+0000 7f1f56973640 -1 osd.1 0 log_to_monitors true 2024-02-10T01:01:42.913 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:01:42.912+0000 7febd0d9e640 -1 osd.0 0 log_to_monitors true 2024-02-10T01:01:42.940 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:01:42.939+0000 7fe730a0f640 -1 osd.2 0 log_to_monitors true 2024-02-10T01:01:45.781 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:01:45.780+0000 7fe72caf9640 -1 osd.2 0 waiting for initial osdmap 2024-02-10T01:01:45.796 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:01:45.795+0000 7fe724105640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T01:01:45.813 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:01:45.812+0000 7f1f52909640 -1 osd.1 0 waiting for initial osdmap 2024-02-10T01:01:45.816 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:01:45.815+0000 7febccd34640 -1 osd.0 0 waiting for initial osdmap 2024-02-10T01:01:45.819 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-02-10T01:01:45.826 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:01:45.825+0000 7f1f4a729640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T01:01:45.831 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:01:45.830+0000 7febc4b54640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T01:01:46.101 INFO:teuthology.misc.health.smithi060.stdout: 2024-02-10T01:01:46.101 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":7,"fsid":"180b9984-42b1-4d69-a091-ca59eeed2339","created":"2024-02-10T01:01:37.970961+0000","modified":"2024-02-10T01:01:44.507046+0000","last_up_change":"0.000000","last_in_change":"2024-02-10T01:01:38.831239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f191b760-bfd1-4937-a959-47327a85bab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d643becc-6621-49bb-9fe8-ae343e7a6317","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:46.112 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-10T01:01:52.415 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-02-10T01:01:52.714 INFO:teuthology.misc.health.smithi060.stdout: 2024-02-10T01:01:52.714 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"180b9984-42b1-4d69-a091-ca59eeed2339","created":"2024-02-10T01:01:37.970961+0000","modified":"2024-02-10T01:01:49.784884+0000","last_up_change":"2024-02-10T01:01:46.782636+0000","last_in_change":"2024-02-10T01:01:38.831239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T01:01:47.546752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6801","nonce":2541152578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6805","nonce":2541152578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6814","nonce":2541152578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6809","nonce":2541152578}]},"public_addr":"172.21.15.60:6801/2541152578","cluster_addr":"172.21.15.60:6805/2541152578","heartbeat_back_addr":"172.21.15.60:6814/2541152578","heartbeat_front_addr":"172.21.15.60:6809/2541152578","state":["exists","up"]},{"osd":1,"uuid":"f191b760-bfd1-4937-a959-47327a85bab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6804","nonce":1726286902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6808","nonce":1726286902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6815","nonce":1726286902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6811","nonce":1726286902}]},"public_addr":"172.21.15.60:6804/1726286902","cluster_addr":"172.21.15.60:6808/1726286902","heartbeat_back_addr":"172.21.15.60:6815/1726286902","heartbeat_front_addr":"172.21.15.60:6811/1726286902","state":["exists","up"]},{"osd":2,"uuid":"d643becc-6621-49bb-9fe8-ae343e7a6317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6817","nonce":956754891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6819","nonce":956754891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6823","nonce":956754891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6821","nonce":956754891}]},"public_addr":"172.21.15.60:6817/956754891","cluster_addr":"172.21.15.60:6819/956754891","heartbeat_back_addr":"172.21.15.60:6823/956754891","heartbeat_front_addr":"172.21.15.60:6821/956754891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.893423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.916962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.973760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:52.726 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-02-10T01:01:52.726 INFO:tasks.ceph:Creating RBD pool 2024-02-10T01:01:52.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-02-10T01:01:53.947 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2024-02-10T01:01:53.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-10T01:01:54.948 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-10T01:01:54.969 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-02-10T01:01:54.969 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-02-10T01:01:54.969 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-02-10T01:01:54.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-02-10T01:01:54.970 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-10T01:01:55.317 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:55.329 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":2837670615},{"type":"v1","addr":"172.21.15.8:6827","nonce":2837670615}]},"active_addr":"172.21.15.8:6827/2837670615","active_change":"2024-02-10T01:01:41.501683+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":903782922}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":581225413}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":878089616}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":2247867515}]}]} 2024-02-10T01:01:55.331 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-02-10T01:01:55.331 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-02-10T01:01:55.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-02-10T01:01:55.625 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:55.625 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"fsid":"180b9984-42b1-4d69-a091-ca59eeed2339","created":"2024-02-10T01:01:37.970961+0000","modified":"2024-02-10T01:01:54.947285+0000","last_up_change":"2024-02-10T01:01:46.782636+0000","last_in_change":"2024-02-10T01:01:38.831239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T01:01:47.546752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-10T01:01:53.036943+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6801","nonce":2541152578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6805","nonce":2541152578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6814","nonce":2541152578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6809","nonce":2541152578}]},"public_addr":"172.21.15.60:6801/2541152578","cluster_addr":"172.21.15.60:6805/2541152578","heartbeat_back_addr":"172.21.15.60:6814/2541152578","heartbeat_front_addr":"172.21.15.60:6809/2541152578","state":["exists","up"]},{"osd":1,"uuid":"f191b760-bfd1-4937-a959-47327a85bab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6804","nonce":1726286902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6808","nonce":1726286902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6815","nonce":1726286902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6811","nonce":1726286902}]},"public_addr":"172.21.15.60:6804/1726286902","cluster_addr":"172.21.15.60:6808/1726286902","heartbeat_back_addr":"172.21.15.60:6815/1726286902","heartbeat_front_addr":"172.21.15.60:6811/1726286902","state":["exists","up"]},{"osd":2,"uuid":"d643becc-6621-49bb-9fe8-ae343e7a6317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6817","nonce":956754891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6819","nonce":956754891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6823","nonce":956754891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6821","nonce":956754891}]},"public_addr":"172.21.15.60:6817/956754891","cluster_addr":"172.21.15.60:6819/956754891","heartbeat_back_addr":"172.21.15.60:6823/956754891","heartbeat_front_addr":"172.21.15.60:6821/956754891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.893423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.916962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.973760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:55.638 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-02-10T01:01:55.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-02-10T01:01:55.930 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:55.930 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"fsid":"180b9984-42b1-4d69-a091-ca59eeed2339","created":"2024-02-10T01:01:37.970961+0000","modified":"2024-02-10T01:01:54.947285+0000","last_up_change":"2024-02-10T01:01:46.782636+0000","last_in_change":"2024-02-10T01:01:38.831239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T01:01:47.546752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-10T01:01:53.036943+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7acfcdc-6d3f-4abf-a6ee-b61f6b49044e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6801","nonce":2541152578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6805","nonce":2541152578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6814","nonce":2541152578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":2541152578},{"type":"v1","addr":"172.21.15.60:6809","nonce":2541152578}]},"public_addr":"172.21.15.60:6801/2541152578","cluster_addr":"172.21.15.60:6805/2541152578","heartbeat_back_addr":"172.21.15.60:6814/2541152578","heartbeat_front_addr":"172.21.15.60:6809/2541152578","state":["exists","up"]},{"osd":1,"uuid":"f191b760-bfd1-4937-a959-47327a85bab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6804","nonce":1726286902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6808","nonce":1726286902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6815","nonce":1726286902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1726286902},{"type":"v1","addr":"172.21.15.60:6811","nonce":1726286902}]},"public_addr":"172.21.15.60:6804/1726286902","cluster_addr":"172.21.15.60:6808/1726286902","heartbeat_back_addr":"172.21.15.60:6815/1726286902","heartbeat_front_addr":"172.21.15.60:6811/1726286902","state":["exists","up"]},{"osd":2,"uuid":"d643becc-6621-49bb-9fe8-ae343e7a6317","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6817","nonce":956754891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6819","nonce":956754891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6823","nonce":956754891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":956754891},{"type":"v1","addr":"172.21.15.60:6821","nonce":956754891}]},"public_addr":"172.21.15.60:6817/956754891","cluster_addr":"172.21.15.60:6819/956754891","heartbeat_back_addr":"172.21.15.60:6823/956754891","heartbeat_front_addr":"172.21.15.60:6821/956754891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.893423+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.916962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T01:01:43.973760+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T01:01:55.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-02-10T01:01:55.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-02-10T01:01:55.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-02-10T01:01:56.084 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-02-10T01:01:56.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-02-10T01:01:56.086 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-02-10T01:01:56.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-02-10T01:01:56.087 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2024-02-10T01:01:56.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-02-10T01:01:56.424 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-02-10T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-02-10T01:01:56.437 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-02-10T01:01:56.440 INFO:teuthology.orchestra.run.smithi060.stdout:34359738370 2024-02-10T01:01:56.449 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-02-10T01:01:56.453 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2024-02-10T01:01:57.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-02-10T01:01:57.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-02-10T01:01:57.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-02-10T01:01:57.762 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-02-10T01:01:57.775 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-02-10T01:01:57.792 INFO:teuthology.orchestra.run.smithi060.stdout:34359738370 2024-02-10T01:01:57.805 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2024-02-10T01:01:57.805 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-02-10T01:01:57.818 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-02-10T01:01:58.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-02-10T01:01:58.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-02-10T01:01:58.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-02-10T01:01:59.104 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-02-10T01:01:59.117 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-02-10T01:01:59.117 DEBUG:teuthology.parallel:result is None 2024-02-10T01:01:59.133 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-02-10T01:01:59.144 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2024-02-10T01:01:59.146 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2024-02-10T01:01:59.146 DEBUG:teuthology.parallel:result is None 2024-02-10T01:01:59.157 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.0 2024-02-10T01:01:59.157 DEBUG:teuthology.parallel:result is None 2024-02-10T01:01:59.157 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-02-10T01:01:59.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-02-10T01:01:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-02-10T01:01:59.498 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-02-10T01:01:57.507669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95044,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020476,"statfs":{"total":289910292480,"available":289812967424,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000676"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.957838+0000","last_change":"2024-02-10T01:01:54.956587+0000","last_active":"2024-02-10T01:01:54.957838+0000","last_peered":"2024-02-10T01:01:54.957838+0000","last_clean":"2024-02-10T01:01:54.957838+0000","last_became_active":"2024-02-10T01:01:54.956286+0000","last_became_peered":"2024-02-10T01:01:54.956286+0000","last_unstale":"2024-02-10T01:01:54.957838+0000","last_undegraded":"2024-02-10T01:01:54.957838+0000","last_fullsized":"2024-02-10T01:01:54.957838+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:51:53.600101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.957444+0000","last_change":"2024-02-10T01:01:54.955705+0000","last_active":"2024-02-10T01:01:54.957444+0000","last_peered":"2024-02-10T01:01:54.957444+0000","last_clean":"2024-02-10T01:01:54.957444+0000","last_became_active":"2024-02-10T01:01:54.955354+0000","last_became_peered":"2024-02-10T01:01:54.955354+0000","last_unstale":"2024-02-10T01:01:54.957444+0000","last_undegraded":"2024-02-10T01:01:54.957444+0000","last_fullsized":"2024-02-10T01:01:54.957444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:57:49.755845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.957875+0000","last_change":"2024-02-10T01:01:54.956628+0000","last_active":"2024-02-10T01:01:54.957875+0000","last_peered":"2024-02-10T01:01:54.957875+0000","last_clean":"2024-02-10T01:01:54.957875+0000","last_became_active":"2024-02-10T01:01:54.956316+0000","last_became_peered":"2024-02-10T01:01:54.956316+0000","last_unstale":"2024-02-10T01:01:54.957875+0000","last_undegraded":"2024-02-10T01:01:54.957875+0000","last_fullsized":"2024-02-10T01:01:54.957875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:06:01.476048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.958556+0000","last_change":"2024-02-10T01:01:54.957117+0000","last_active":"2024-02-10T01:01:54.958556+0000","last_peered":"2024-02-10T01:01:54.958556+0000","last_clean":"2024-02-10T01:01:54.958556+0000","last_became_active":"2024-02-10T01:01:54.956768+0000","last_became_peered":"2024-02-10T01:01:54.956768+0000","last_unstale":"2024-02-10T01:01:54.958556+0000","last_undegraded":"2024-02-10T01:01:54.958556+0000","last_fullsized":"2024-02-10T01:01:54.958556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:16:34.021120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:55.955610+0000","last_change":"2024-02-10T01:01:54.958609+0000","last_active":"2024-02-10T01:01:55.955610+0000","last_peered":"2024-02-10T01:01:55.955610+0000","last_clean":"2024-02-10T01:01:55.955610+0000","last_became_active":"2024-02-10T01:01:54.958286+0000","last_became_peered":"2024-02-10T01:01:54.958286+0000","last_unstale":"2024-02-10T01:01:55.955610+0000","last_undegraded":"2024-02-10T01:01:55.955610+0000","last_fullsized":"2024-02-10T01:01:55.955610+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:10:05.591067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.957992+0000","last_change":"2024-02-10T01:01:54.955976+0000","last_active":"2024-02-10T01:01:54.957992+0000","last_peered":"2024-02-10T01:01:54.957992+0000","last_clean":"2024-02-10T01:01:54.957992+0000","last_became_active":"2024-02-10T01:01:54.955626+0000","last_became_peered":"2024-02-10T01:01:54.955626+0000","last_unstale":"2024-02-10T01:01:54.957992+0000","last_undegraded":"2024-02-10T01:01:54.957992+0000","last_fullsized":"2024-02-10T01:01:54.957992+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:46:57.322314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.957953+0000","last_change":"2024-02-10T01:01:54.957065+0000","last_active":"2024-02-10T01:01:54.957953+0000","last_peered":"2024-02-10T01:01:54.957953+0000","last_clean":"2024-02-10T01:01:54.957953+0000","last_became_active":"2024-02-10T01:01:54.956742+0000","last_became_peered":"2024-02-10T01:01:54.956742+0000","last_unstale":"2024-02-10T01:01:54.957953+0000","last_undegraded":"2024-02-10T01:01:54.957953+0000","last_fullsized":"2024-02-10T01:01:54.957953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:18:46.962433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.957402+0000","last_change":"2024-02-10T01:01:54.955839+0000","last_active":"2024-02-10T01:01:54.957402+0000","last_peered":"2024-02-10T01:01:54.957402+0000","last_clean":"2024-02-10T01:01:54.957402+0000","last_became_active":"2024-02-10T01:01:54.955044+0000","last_became_peered":"2024-02-10T01:01:54.955044+0000","last_unstale":"2024-02-10T01:01:54.957402+0000","last_undegraded":"2024-02-10T01:01:54.957402+0000","last_fullsized":"2024-02-10T01:01:54.957402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:40:26.376852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-02-10T01:01:54.953728+0000","last_change":"2024-02-10T01:01:48.931330+0000","last_active":"2024-02-10T01:01:54.953728+0000","last_peered":"2024-02-10T01:01:54.953728+0000","last_clean":"2024-02-10T01:01:54.953728+0000","last_became_active":"2024-02-10T01:01:48.931017+0000","last_became_peered":"2024-02-10T01:01:48.931017+0000","last_unstale":"2024-02-10T01:01:54.953728+0000","last_undegraded":"2024-02-10T01:01:54.953728+0000","last_fullsized":"2024-02-10T01:01:54.953728+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:47.783329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:47.783329+0000","last_clean_scrub_stamp":"2024-02-10T01:01:47.783329+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:11.064705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T01:01:59.499 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-02-10T01:01:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:01:59.790 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-02-10T01:01:59.801 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-02-10T01:01:59.508209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95060,"kb_used_data":1512,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020460,"statfs":{"total":289910292480,"available":289812951040,"internally_reserved":0,"allocated":1548288,"data_stored":1281757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001216"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.817017+0000","last_change":"2024-02-10T01:01:54.956587+0000","last_active":"2024-02-10T01:01:56.817017+0000","last_peered":"2024-02-10T01:01:56.817017+0000","last_clean":"2024-02-10T01:01:56.817017+0000","last_became_active":"2024-02-10T01:01:54.956286+0000","last_became_peered":"2024-02-10T01:01:54.956286+0000","last_unstale":"2024-02-10T01:01:56.817017+0000","last_undegraded":"2024-02-10T01:01:56.817017+0000","last_fullsized":"2024-02-10T01:01:56.817017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:32:02.658019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.817013+0000","last_change":"2024-02-10T01:01:54.955705+0000","last_active":"2024-02-10T01:01:56.817013+0000","last_peered":"2024-02-10T01:01:56.817013+0000","last_clean":"2024-02-10T01:01:56.817013+0000","last_became_active":"2024-02-10T01:01:54.955354+0000","last_became_peered":"2024-02-10T01:01:54.955354+0000","last_unstale":"2024-02-10T01:01:56.817013+0000","last_undegraded":"2024-02-10T01:01:56.817013+0000","last_fullsized":"2024-02-10T01:01:56.817013+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:30:51.476201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.817076+0000","last_change":"2024-02-10T01:01:54.956628+0000","last_active":"2024-02-10T01:01:56.817076+0000","last_peered":"2024-02-10T01:01:56.817076+0000","last_clean":"2024-02-10T01:01:56.817076+0000","last_became_active":"2024-02-10T01:01:54.956316+0000","last_became_peered":"2024-02-10T01:01:54.956316+0000","last_unstale":"2024-02-10T01:01:56.817076+0000","last_undegraded":"2024-02-10T01:01:56.817076+0000","last_fullsized":"2024-02-10T01:01:56.817076+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:46:08.240524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.816989+0000","last_change":"2024-02-10T01:01:54.957117+0000","last_active":"2024-02-10T01:01:56.816989+0000","last_peered":"2024-02-10T01:01:56.816989+0000","last_clean":"2024-02-10T01:01:56.816989+0000","last_became_active":"2024-02-10T01:01:54.956768+0000","last_became_peered":"2024-02-10T01:01:54.956768+0000","last_unstale":"2024-02-10T01:01:56.816989+0000","last_undegraded":"2024-02-10T01:01:56.816989+0000","last_fullsized":"2024-02-10T01:01:56.816989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:45:57.956365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:55.955610+0000","last_change":"2024-02-10T01:01:54.958609+0000","last_active":"2024-02-10T01:01:55.955610+0000","last_peered":"2024-02-10T01:01:55.955610+0000","last_clean":"2024-02-10T01:01:55.955610+0000","last_became_active":"2024-02-10T01:01:54.958286+0000","last_became_peered":"2024-02-10T01:01:54.958286+0000","last_unstale":"2024-02-10T01:01:55.955610+0000","last_undegraded":"2024-02-10T01:01:55.955610+0000","last_fullsized":"2024-02-10T01:01:55.955610+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:10:05.591067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.343669+0000","last_change":"2024-02-10T01:01:54.955976+0000","last_active":"2024-02-10T01:01:56.343669+0000","last_peered":"2024-02-10T01:01:56.343669+0000","last_clean":"2024-02-10T01:01:56.343669+0000","last_became_active":"2024-02-10T01:01:54.955626+0000","last_became_peered":"2024-02-10T01:01:54.955626+0000","last_unstale":"2024-02-10T01:01:56.343669+0000","last_undegraded":"2024-02-10T01:01:56.343669+0000","last_fullsized":"2024-02-10T01:01:56.343669+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:28.964489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.343639+0000","last_change":"2024-02-10T01:01:54.957065+0000","last_active":"2024-02-10T01:01:56.343639+0000","last_peered":"2024-02-10T01:01:56.343639+0000","last_clean":"2024-02-10T01:01:56.343639+0000","last_became_active":"2024-02-10T01:01:54.956742+0000","last_became_peered":"2024-02-10T01:01:54.956742+0000","last_unstale":"2024-02-10T01:01:56.343639+0000","last_undegraded":"2024-02-10T01:01:56.343639+0000","last_fullsized":"2024-02-10T01:01:56.343639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:02:20.756166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.816979+0000","last_change":"2024-02-10T01:01:54.955839+0000","last_active":"2024-02-10T01:01:56.816979+0000","last_peered":"2024-02-10T01:01:56.816979+0000","last_clean":"2024-02-10T01:01:56.816979+0000","last_became_active":"2024-02-10T01:01:54.955044+0000","last_became_peered":"2024-02-10T01:01:54.955044+0000","last_unstale":"2024-02-10T01:01:56.816979+0000","last_undegraded":"2024-02-10T01:01:56.816979+0000","last_fullsized":"2024-02-10T01:01:56.816979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:53.944845+0000","last_clean_scrub_stamp":"2024-02-10T01:01:53.944845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:27:58.598402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-02-10T01:01:56.816960+0000","last_change":"2024-02-10T01:01:48.931330+0000","last_active":"2024-02-10T01:01:56.816960+0000","last_peered":"2024-02-10T01:01:56.816960+0000","last_clean":"2024-02-10T01:01:56.816960+0000","last_became_active":"2024-02-10T01:01:48.931017+0000","last_became_peered":"2024-02-10T01:01:48.931017+0000","last_unstale":"2024-02-10T01:01:56.816960+0000","last_undegraded":"2024-02-10T01:01:56.816960+0000","last_fullsized":"2024-02-10T01:01:56.816960+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T01:01:47.783329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T01:01:47.783329+0000","last_clean_scrub_stamp":"2024-02-10T01:01:47.783329+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:11.064705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":122880,"data_stored":34833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":625201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T01:01:59.802 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-02-10T01:01:59.803 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-02-10T01:01:59.803 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-02-10T01:01:59.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-02-10T01:02:00.132 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-02-10T01:02:00.132 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T01:02:00.145 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-02-10T01:02:00.145 INFO:teuthology.run_tasks:Running task exec... 2024-02-10T01:02:00.156 INFO:teuthology.task.exec:Executing custom commands... 2024-02-10T01:02:00.156 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T01:02:00.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:00.541 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror] 2024-02-10T01:02:00.541 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDCysZliFGaFhAAjspKwHkrVIf/vGGGYikX0A== 2024-02-10T01:02:00.541 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror-peer" 2024-02-10T01:02:00.541 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:00.541 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror 2024-02-10T01:02:00.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:00.930 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-02-10T01:02:00.930 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDCysZlom4EGhAAdBeqgw3ypT2+2XHI1ZEFMA== 2024-02-10T01:02:00.930 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:00.930 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:00.930 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.0 2024-02-10T01:02:00.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:01.315 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-02-10T01:02:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDCysZl3hXTHxAAl05Zb3Tf9ovUjUUAeAdbKA== 2024-02-10T01:02:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.1 2024-02-10T01:02:01.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:01.709 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-02-10T01:02:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDCysZlbcOVJRAA6iCLpY7F75g5EH7eYCPpkQ== 2024-02-10T01:02:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:01.710 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.2 2024-02-10T01:02:01.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:02.102 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-02-10T01:02:02.102 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDCysZl0HVkKxAAMRQECJtGofnYeQi5uKeqVw== 2024-02-10T01:02:02.102 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:02.103 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.3 2024-02-10T01:02:02.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror] 2024-02-10T01:02:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlrQ/zIhAARkNjGy5uc78exGkHoaQ6lg== 2024-02-10T01:02:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror-peer" 2024-02-10T01:02:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror 2024-02-10T01:02:02.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:02.876 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-02-10T01:02:02.876 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlxkmxKBAASQBHlH/YKgmfr6zNF3ue1A== 2024-02-10T01:02:02.876 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:02.876 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:02.876 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.0 2024-02-10T01:02:02.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:03.266 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-02-10T01:02:03.267 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlNgt4LhAAAMNWEsGpZBAv4nBuSIw8gw== 2024-02-10T01:02:03.267 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:03.267 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:03.267 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.1 2024-02-10T01:02:03.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:03.657 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-02-10T01:02:03.657 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlUHY/NBAA4c2z+aYdl5Wo1CwRksNZXA== 2024-02-10T01:02:03.658 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:03.658 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:03.658 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.2 2024-02-10T01:02:03.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-02-10T01:02:04.044 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-02-10T01:02:04.044 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDlysZlWuYaOhAABpG1AD0Zsog6h2T7oP6TAg== 2024-02-10T01:02:04.044 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-02-10T01:02:04.044 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-02-10T01:02:04.045 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.3 2024-02-10T01:02:04.057 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:02:04.093 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:02:04.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:02:04.096 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-02-10T01:02:04.109 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-02-10T01:02:04.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-02-10T01:02:04.111 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-02-10T01:02:04.137 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-02-10T01:02:04.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-02-10T01:02:04.139 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-02-10T01:02:04.178 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-02-10T01:02:04.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-02-10T01:02:04.181 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-02-10T01:02:04.211 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-02-10T01:02:04.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-02-10T01:02:04.214 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-02-10T01:02:04.245 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-02-10T01:02:04.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-02-10T01:02:04.247 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-02-10T01:02:04.288 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-02-10T01:02:04.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-02-10T01:02:04.290 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-02-10T01:02:04.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-02-10T01:02:04.329 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-02-10T01:02:04.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-02-10T01:02:04.331 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-02-10T01:02:04.397 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1707526924 2024-02-10T01:02:04.398 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-02-10T01:02:04.398 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-02-10T01:02:04.412 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1707526924 2024-02-10T01:02:04.412 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-02-10T01:02:04.412 INFO:teuthology.run_tasks:Running task workunit... 2024-02-10T01:02:04.458 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T01:02:04.458 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-10T01:02:04.458 DEBUG:teuthology.orchestra.run.smithi060:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-02-10T01:02:04.459 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-02-10T01:02:04.459 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-02-10T01:02:04.459 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-02-10T01:02:04.459 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-02-10T01:02:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T01:02:04.474 INFO:teuthology.orchestra.run.smithi060.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-02-10T01:02:04.475 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-02-10T01:02:04.529 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-02-10T01:02:04.530 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-02-10T01:02:04.584 INFO:tasks.workunit:timeout=6h 2024-02-10T01:02:04.584 INFO:tasks.workunit:cleanup=True 2024-02-10T01:02:04.585 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T01:02:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-02-10T01:03:25.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5488108349510303) > weight (0.25) 2024-02-10T01:03:25.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9402109212221308) > weight (0.25) 2024-02-10T01:03:25.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8577582902779586) > weight (0.25) 2024-02-10T01:03:25.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.283344217746871) > weight (0.25) 2024-02-10T01:03:25.461 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-02-10T01:03:40.460 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-02-10T01:03:40.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-02-10T01:03:40.462 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-02-10T01:03:40.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:2024-02-10T01:03:40.461+0000 7f02b9965640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38756) UID: 1000 2024-02-10T01:03:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Updating files: 95% (11611/12104) Updating files: 96% (11620/12104) Updating files: 97% (11741/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-02-10T01:03:45.463 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-02-10T01:03:45.463 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:03:45.463 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-02-10T01:03:45.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-02-10T01:03:45.464 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-02-10T01:03:45.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-02-10T01:03:45.466 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-02-10T01:03:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-10T01:03:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: git switch -c 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Or undo this operation with: 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: git switch - 2024-02-10T01:03:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T01:03:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:03:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-10T01:03:45.803 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-02-10T01:03:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T01:03:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-02-10T01:03:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T01:03:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T01:03:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T01:03:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-02-10T01:03:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-02-10T01:03:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T01:03:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-02-10T01:03:46.817 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-02-10T01:03:46.817 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-02-10T01:03:46.872 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-02-10T01:03:46.873 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-02-10T01:03:46.873 DEBUG:teuthology.orchestra.run.smithi060:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-02-10T01:03:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMAGE_COUNT=50 2024-02-10T01:03:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ export LOCKDEP=0 2024-02-10T01:03:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ LOCKDEP=0 2024-02-10T01:03:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-02-10T01:03:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-02-10T01:03:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ type xmlstarlet 2024-02-10T01:03:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ XMLSTARLET=xmlstarlet 2024-02-10T01:03:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ RBD_MIRROR_INSTANCES=4 2024-02-10T01:03:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CLUSTER1=cluster1 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CLUSTER2=cluster2 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ PEER_CLUSTER_SUFFIX= 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ POOL=mirror 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ PARENT_POOL=mirror_parent 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ NS1=ns1 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ NS2=ns2 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ TEMPDIR= 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ID=mirror 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ RBD_IMAGE_FEATURES=layering 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ RBD_MIRROR_MODE=snapshot 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ MIRROR_POOL_MODE=image 2024-02-10T01:03:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' snapshot = snapshot ']' 2024-02-10T01:03:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ MIRROR_POOL_MODE=image 2024-02-10T01:03:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-02-10T01:03:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS='--id mirror' 2024-02-10T01:03:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ LAST_MIRROR_INSTANCE=3 2024-02-10T01:03:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-02-10T01:03:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-02-10T01:03:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_BIN=. 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_SRC=. 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' -e CMakeCache.txt ']' 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 -gt 0 ']' 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local c 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup_tempdir 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mktemp -d 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ TEMPDIR=/tmp/tmp.ywJfcqGoUP 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z 1 ']' 2024-02-10T01:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup_pools cluster1 cluster2 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster2 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_map_file 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_addr 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local admin_key_file 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-02-10T01:03:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror' already exists 2024-02-10T01:03:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-02-10T01:03:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror_parent' already exists 2024-02-10T01:03:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 pool init mirror 2024-02-10T01:03:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-02-10T01:03:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:03:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-02-10T01:03:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-02-10T01:03:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-02-10T01:03:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-02-10T01:03:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-02-10T01:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-02-10T01:03:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster1 mirror cluster2 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster2 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster2 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-02-10T01:03:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-02-10T01:03:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-02-10T01:03:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:03:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=2dc2868b-b7dc-4646-98ea-74d206deca06 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror_parent 2024-02-10T01:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster2 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster2 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=2b77c1ab-96e8-4c99-b228-f5be0d8b4938 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup_pools cluster2 cluster1 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster1 2024-02-10T01:03:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_map_file 2024-02-10T01:03:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_addr 2024-02-10T01:03:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local admin_key_file 2024-02-10T01:03:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid 2024-02-10T01:03:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-02-10T01:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror' already exists 2024-02-10T01:03:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-02-10T01:03:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror_parent' already exists 2024-02-10T01:03:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:03:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 pool init mirror 2024-02-10T01:04:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:04:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-02-10T01:04:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:04:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-02-10T01:04:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-02-10T01:04:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-02-10T01:04:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-02-10T01:04:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-02-10T01:04:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-02-10T01:04:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-02-10T01:04:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-02-10T01:04:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster2 mirror cluster1 2024-02-10T01:04:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster1 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster1 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:04:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=d3038c02-eb34-4a86-b8ac-81ae1e74b5d6 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror_parent 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster1 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster1 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:04:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-02-10T01:04:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=6cd86f4e-93d0-43ac-a643-acec33df4c63 2024-02-10T01:04:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-02-10T01:04:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirrors cluster1 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 3 2024-02-10T01:04:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster1:0 2024-02-10T01:04:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:0 2024-02-10T01:04:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster1:0 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster1 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=0 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster1 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=0 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster1:1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster1:1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 1 = cluster1:1 ']' 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=1 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster1:2 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:2 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster1:2 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster1 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=2 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 2 = cluster1:2 ']' 2024-02-10T01:04:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster1 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=2 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster1:3 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:3 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster1:3 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster1 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=3 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 3 = cluster1:3 ']' 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster1 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=3 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirrors cluster2 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 3 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster2:0 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:0 2024-02-10T01:04:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster2:0 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster2 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=0 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 = cluster2:0 ']' 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster2 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=0 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster2:1 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:1 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster2:1 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster2 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=1 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 1 = cluster2:1 ']' 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster2 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=1 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:04:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster2:2 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:2 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster2:2 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster2 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=2 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 2 = cluster2:2 ']' 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster2 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=2 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster2:3 2024-02-10T01:04:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:3 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster2:3 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster2 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=3 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 3 = cluster2:3 ']' 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster2 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=3 2024-02-10T01:04:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee -a /tmp/tmp.ywJfcqGoUP/rbd-mirror.test.log 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ date '+%F %T' 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10 01:04:03 TEST: add image and test replay after client crashes 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10 01:04:03 TEST: add image and test replay after client crashes 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=test 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror test 512M 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 512M ']' 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=512M 2024-02-10T01:04:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:04:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2024-02-10T01:04:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:04:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:04:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-02-10T01:04:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:04:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10ae2e584144 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10ae2e584144 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:04:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:03 2024 2024-02-10T01:04:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:04:03 2024 2024-02-10T01:04:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:03 2024 2024-02-10T01:04:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:04:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:04:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:04:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:04:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:04:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:04:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:04:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:04:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:04:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:04:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:04:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:04:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:04:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:04:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:04:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:04:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:04:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:04:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:04:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:04:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:04:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:04:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:04:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:04:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:04:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:04:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:04:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:04:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:04:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:04:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:04:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:04:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:04:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:04:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:04:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:04:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:04:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:04:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:04:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:04:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:04:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:04:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:04:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:04:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 1 2024-02-10T01:04:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:04:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:04:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:04:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:04:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:04:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:04:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:04:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:04:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:04:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:04:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:04:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:04:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:04:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name= 2024-02-10T01:04:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 10 2024-02-10T01:04:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:04:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:04:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:04:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:04:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:04:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:04:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 11743232 bytes 2024-02-10T01:04:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 60829696 bytes 2024-02-10T01:04:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 93175808 bytes 2024-02-10T01:04:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 125340672 bytes 2024-02-10T01:04:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 175262720 bytes 2024-02-10T01:04:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 213742592 bytes 2024-02-10T01:04:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 227324928 bytes 2024-02-10T01:04:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 281282560 bytes 2024-02-10T01:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 324877312 bytes 2024-02-10T01:04:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 411902976 bytes 2024-02-10T01:04:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 442318848 bytes 2024-02-10T01:04:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 504642560 bytes 2024-02-10T01:04:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 520072192 bytes 2024-02-10T01:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 595552256 bytes 2024-02-10T01:04:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 623986688 bytes 2024-02-10T01:04:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 637642752 bytes 2024-02-10T01:04:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 692940800 bytes 2024-02-10T01:04:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 745090048 bytes 2024-02-10T01:04:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 837501952 bytes 2024-02-10T01:04:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 860650496 bytes 2024-02-10T01:04:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 953874432 bytes 2024-02-10T01:04:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1044246528 bytes 2024-02-10T01:04:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1081772032 bytes 2024-02-10T01:04:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1174747136 bytes 2024-02-10T01:04:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1223057408 bytes 2024-02-10T01:04:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1247804416 bytes 2024-02-10T01:04:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1337802752 bytes 2024-02-10T01:04:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1354343424 bytes 2024-02-10T01:04:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1384424448 bytes 2024-02-10T01:04:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1476451328 bytes 2024-02-10T01:04:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1509054464 bytes 2024-02-10T01:04:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1553989632 bytes 2024-02-10T01:04:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1613122560 bytes 2024-02-10T01:04:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1620373504 bytes 2024-02-10T01:04:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1647279104 bytes 2024-02-10T01:04:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1722699776 bytes 2024-02-10T01:04:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1756681216 bytes 2024-02-10T01:04:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1782433792 bytes 2024-02-10T01:04:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1834683392 bytes 2024-02-10T01:04:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1896753152 bytes 2024-02-10T01:04:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1968088064 bytes 2024-02-10T01:04:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 2010778624 bytes 2024-02-10T01:04:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2043129856 bytes 2024-02-10T01:04:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2065219584 bytes 2024-02-10T01:04:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2101753856 bytes 2024-02-10T01:04:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2161136640 bytes 2024-02-10T01:04:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2206099456 bytes 2024-02-10T01:04:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2269234176 bytes 2024-02-10T01:04:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2309619712 bytes 2024-02-10T01:04:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2395898880 bytes 2024-02-10T01:04:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2413062144 bytes 2024-02-10T01:04:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2441404416 bytes 2024-02-10T01:04:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2462094336 bytes 2024-02-10T01:04:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2562205696 bytes 2024-02-10T01:04:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2596344832 bytes 2024-02-10T01:04:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2637566976 bytes 2024-02-10T01:04:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2646569984 bytes 2024-02-10T01:04:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2675872768 bytes 2024-02-10T01:04:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2785780736 bytes 2024-02-10T01:04:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2843452416 bytes 2024-02-10T01:04:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2844500992 bytes 2024-02-10T01:04:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2891098112 bytes 2024-02-10T01:04:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2975676416 bytes 2024-02-10T01:04:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 3010666496 bytes 2024-02-10T01:04:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 3049089024 bytes 2024-02-10T01:04:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3073483776 bytes 2024-02-10T01:04:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3117466624 bytes 2024-02-10T01:04:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3166014464 bytes 2024-02-10T01:04:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3193101312 bytes 2024-02-10T01:04:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3216434176 bytes 2024-02-10T01:04:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3247579136 bytes 2024-02-10T01:04:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3316317184 bytes 2024-02-10T01:04:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3368433664 bytes 2024-02-10T01:04:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3450873856 bytes 2024-02-10T01:04:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3521114112 bytes 2024-02-10T01:04:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3545972736 bytes 2024-02-10T01:04:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3603451904 bytes 2024-02-10T01:04:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3638938624 bytes 2024-02-10T01:04:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3694984192 bytes 2024-02-10T01:04:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3739294720 bytes 2024-02-10T01:04:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3794663424 bytes 2024-02-10T01:04:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3851058176 bytes 2024-02-10T01:04:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3935042560 bytes 2024-02-10T01:04:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3956131840 bytes 2024-02-10T01:04:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 4025960448 bytes 2024-02-10T01:04:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 4077400064 bytes 2024-02-10T01:04:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 4135801856 bytes 2024-02-10T01:04:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 4198881280 bytes 2024-02-10T01:04:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 4236185600 bytes 2024-02-10T01:04:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 4286610432 bytes 2024-02-10T01:04:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 4382448640 bytes 2024-02-10T01:04:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 4430367744 bytes 2024-02-10T01:04:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 4486659072 bytes 2024-02-10T01:04:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 4558854144 bytes 2024-02-10T01:04:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 4559452160 bytes 2024-02-10T01:04:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 4622412800 bytes 2024-02-10T01:04:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4668903424 bytes 2024-02-10T01:04:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4703524864 bytes 2024-02-10T01:04:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4726134784 bytes 2024-02-10T01:04:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4751602688 bytes 2024-02-10T01:04:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4830946304 bytes 2024-02-10T01:04:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 4851304448 bytes 2024-02-10T01:04:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 4899301376 bytes 2024-02-10T01:04:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 4904384512 bytes 2024-02-10T01:04:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10500 IOs, 5013497856 bytes 2024-02-10T01:04:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10600 IOs, 5083211776 bytes 2024-02-10T01:04:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10700 IOs, 5118682112 bytes 2024-02-10T01:04:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10800 IOs, 5141036032 bytes 2024-02-10T01:04:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10900 IOs, 5189033984 bytes 2024-02-10T01:04:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11000 IOs, 5225682944 bytes 2024-02-10T01:04:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11100 IOs, 5272467456 bytes 2024-02-10T01:04:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11200 IOs, 5354719232 bytes 2024-02-10T01:04:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11300 IOs, 5426686976 bytes 2024-02-10T01:04:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11400 IOs, 5427312640 bytes 2024-02-10T01:04:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11500 IOs, 5488193536 bytes 2024-02-10T01:04:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11600 IOs, 5547823104 bytes 2024-02-10T01:04:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11700 IOs, 5608408064 bytes 2024-02-10T01:04:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11800 IOs, 5642777600 bytes 2024-02-10T01:04:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11900 IOs, 5659845632 bytes 2024-02-10T01:04:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12000 IOs, 5768609792 bytes 2024-02-10T01:04:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12100 IOs, 5805017088 bytes 2024-02-10T01:04:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12200 IOs, 5841171456 bytes 2024-02-10T01:04:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12300 IOs, 5875932160 bytes 2024-02-10T01:04:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12400 IOs, 5888915456 bytes 2024-02-10T01:04:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12500 IOs, 5945849856 bytes 2024-02-10T01:04:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12600 IOs, 6005182464 bytes 2024-02-10T01:04:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12700 IOs, 6028217344 bytes 2024-02-10T01:04:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12800 IOs, 6082814976 bytes 2024-02-10T01:04:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12900 IOs, 6158266368 bytes 2024-02-10T01:04:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:04:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:04:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:04:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:04:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:04:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:04:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:04:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:04:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:04:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:04:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:04:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:04:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-02-10T01:04:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:04:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:04:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:04:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:04:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:04:22.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:04:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-02-10T01:04:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:04:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:04:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:04:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:04:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:04:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:04:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:04:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:04:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:04:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 8 2024-02-10T01:04:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:04:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:04:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:04:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:04:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:04:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:04:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527044,"remote_snapshot_timestamp":1707527044,"replay_state":"idle"} 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:04:34 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:04:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:04:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:04:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:04:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:04:34 2024-02-10T01:04:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:04:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527044,"remote_snapshot_timestamp":1707527044,"replay_state":"idle"} 2024-02-10T01:04:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:04:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:04:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:04:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:04:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:04:34' 2024-02-10T01:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:04:34' 2024-02-10T01:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:04:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:04:34' 2024-02-10T01:04:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:04:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:04:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:04:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:04:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:04:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:04:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:04:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:04:34' 2024-02-10T01:04:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:04:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:04:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:04:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:04:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap1 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap1 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1 2024-02-10T01:04:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:04:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:04:48.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9111210666583763) > weight (0.25) 2024-02-10T01:04:48.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5413572830391358) > weight (0.25) 2024-02-10T01:04:48.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.622326203506878) > weight (0.25) 2024-02-10T01:04:48.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9661499737702636) > weight (0.25) 2024-02-10T01:04:48.468 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-02-10T01:04:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:04:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:04:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:04:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:04:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:04:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:04:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:04:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:04:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:04:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:04:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:04:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:04:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:04:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:04:49.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.491580367304319) > weight (0.25) 2024-02-10T01:04:49.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8315125323609565) > weight (0.25) 2024-02-10T01:04:49.462 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-02-10T01:04:49.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-02-10T01:04:49.463 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-02-10T01:04:49.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:04:49.463+0000 7fb5737d3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38654) UID: 1000 2024-02-10T01:04:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 5 2024-02-10T01:04:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:04:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:04:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:04:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:04:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:04:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=5 2024-02-10T01:04:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:04:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:04:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:04:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:04:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:04:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:04:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=3 2024-02-10T01:04:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 5 = 3 2024-02-10T01:04:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:04:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:04:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:04:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:04:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:04:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:04:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:04:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:04:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:04:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:04:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 5 = '' 2024-02-10T01:04:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:04:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:04:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:04:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:04:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:04:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:04:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:04:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:04:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:04:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 5 = '' 2024-02-10T01:04:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:04:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:04:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:04:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:04:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:04:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=5 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 5 = 5 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:04:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:04:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:04:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:04:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:04:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap1 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1 2024-02-10T01:04:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:04:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:04:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:04:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:04:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:04:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:04:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:04:55.464 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-02-10T01:04:55.464 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:04:55.465 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-02-10T01:04:55.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-02-10T01:04:55.465 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-02-10T01:04:55.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-02-10T01:04:55.468 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-02-10T01:04:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:04:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:04:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:04:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap1-clean 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1-clean 2024-02-10T01:04:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:04:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:04:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:04:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:04:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:04:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:04:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:04:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:04:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:04:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:04:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 65442816 bytes 2024-02-10T01:04:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 142654464 bytes 2024-02-10T01:04:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 162851840 bytes 2024-02-10T01:04:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 223479808 bytes 2024-02-10T01:04:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 283370496 bytes 2024-02-10T01:04:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 329712640 bytes 2024-02-10T01:04:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 378373120 bytes 2024-02-10T01:04:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 413622272 bytes 2024-02-10T01:04:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 448408576 bytes 2024-02-10T01:04:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 482200576 bytes 2024-02-10T01:04:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 484716544 bytes 2024-02-10T01:04:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 501458944 bytes 2024-02-10T01:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 515584000 bytes 2024-02-10T01:04:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 538137600 bytes 2024-02-10T01:04:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 567089152 bytes 2024-02-10T01:04:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 567713792 bytes 2024-02-10T01:04:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 633610240 bytes 2024-02-10T01:04:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 697440256 bytes 2024-02-10T01:04:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 779850752 bytes 2024-02-10T01:04:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 830420992 bytes 2024-02-10T01:04:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 871631872 bytes 2024-02-10T01:04:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 894945280 bytes 2024-02-10T01:04:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 946161664 bytes 2024-02-10T01:04:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 993118208 bytes 2024-02-10T01:04:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1008679936 bytes 2024-02-10T01:04:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1069660160 bytes 2024-02-10T01:04:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1110131712 bytes 2024-02-10T01:04:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1167813632 bytes 2024-02-10T01:04:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1221988352 bytes 2024-02-10T01:04:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1228525568 bytes 2024-02-10T01:05:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1285154816 bytes 2024-02-10T01:05:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1327963136 bytes 2024-02-10T01:05:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1358507008 bytes 2024-02-10T01:05:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1379977216 bytes 2024-02-10T01:05:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1411698688 bytes 2024-02-10T01:05:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1446599680 bytes 2024-02-10T01:05:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1463093248 bytes 2024-02-10T01:05:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1515019264 bytes 2024-02-10T01:05:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1619948544 bytes 2024-02-10T01:05:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1686342656 bytes 2024-02-10T01:05:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1729664000 bytes 2024-02-10T01:05:01.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1756826624 bytes 2024-02-10T01:05:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1802739712 bytes 2024-02-10T01:05:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1803337728 bytes 2024-02-10T01:05:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1893788672 bytes 2024-02-10T01:05:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1949722624 bytes 2024-02-10T01:05:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1956297728 bytes 2024-02-10T01:05:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1992189952 bytes 2024-02-10T01:05:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2051917824 bytes 2024-02-10T01:05:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2061576192 bytes 2024-02-10T01:05:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2147415040 bytes 2024-02-10T01:05:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2231860224 bytes 2024-02-10T01:05:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2280217600 bytes 2024-02-10T01:05:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2280812544 bytes 2024-02-10T01:05:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2282035200 bytes 2024-02-10T01:05:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2327078912 bytes 2024-02-10T01:05:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2349949952 bytes 2024-02-10T01:05:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2381952000 bytes 2024-02-10T01:05:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2404222976 bytes 2024-02-10T01:05:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2491021312 bytes 2024-02-10T01:05:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2548882432 bytes 2024-02-10T01:05:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2561512448 bytes 2024-02-10T01:05:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2603344896 bytes 2024-02-10T01:05:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2603945984 bytes 2024-02-10T01:05:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2641356800 bytes 2024-02-10T01:05:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2727106560 bytes 2024-02-10T01:05:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2768644096 bytes 2024-02-10T01:05:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2930886656 bytes 2024-02-10T01:05:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3012960256 bytes 2024-02-10T01:05:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3029094400 bytes 2024-02-10T01:05:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3069102080 bytes 2024-02-10T01:05:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3082708992 bytes 2024-02-10T01:05:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3136502784 bytes 2024-02-10T01:05:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3171074048 bytes 2024-02-10T01:05:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3218995200 bytes 2024-02-10T01:05:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3292901376 bytes 2024-02-10T01:05:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3347156992 bytes 2024-02-10T01:05:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3387432960 bytes 2024-02-10T01:05:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3473400832 bytes 2024-02-10T01:05:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3500826624 bytes 2024-02-10T01:05:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3505495040 bytes 2024-02-10T01:05:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3506104320 bytes 2024-02-10T01:05:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3558995968 bytes 2024-02-10T01:05:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3599212544 bytes 2024-02-10T01:05:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3658813440 bytes 2024-02-10T01:05:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3679575040 bytes 2024-02-10T01:05:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3711150080 bytes 2024-02-10T01:05:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3727467520 bytes 2024-02-10T01:05:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3785004032 bytes 2024-02-10T01:05:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3831373824 bytes 2024-02-10T01:05:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3886209024 bytes 2024-02-10T01:05:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3941773312 bytes 2024-02-10T01:05:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3966138368 bytes 2024-02-10T01:05:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3983118336 bytes 2024-02-10T01:05:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3985715200 bytes 2024-02-10T01:05:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3999221760 bytes 2024-02-10T01:05:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4054671360 bytes 2024-02-10T01:05:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4113630208 bytes 2024-02-10T01:05:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4154137600 bytes 2024-02-10T01:05:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4267400192 bytes 2024-02-10T01:05:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4324690944 bytes 2024-02-10T01:05:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 4400847872 bytes 2024-02-10T01:05:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 4439135232 bytes 2024-02-10T01:05:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 4480629760 bytes 2024-02-10T01:05:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10500 IOs, 4508895232 bytes 2024-02-10T01:05:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10600 IOs, 4531579904 bytes 2024-02-10T01:05:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10700 IOs, 4586352640 bytes 2024-02-10T01:05:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10800 IOs, 4628551680 bytes 2024-02-10T01:05:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10900 IOs, 4708243456 bytes 2024-02-10T01:05:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11000 IOs, 4746444800 bytes 2024-02-10T01:05:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11100 IOs, 4754956288 bytes 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:05:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:05:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:05:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:05:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:05:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:05:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:05:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:05:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":17885969.07,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527090,"remote_snapshot_timestamp":1707527090,"replay_state":"idle"} 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:05:04 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:05:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:05:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:05:04 2024-02-10T01:05:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:05:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":17885969.07,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527090,"remote_snapshot_timestamp":1707527090,"replay_state":"idle"} 2024-02-10T01:05:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:05:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:05:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:05:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:05:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:05:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:05:04' 2024-02-10T01:05:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:05:04' 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:05:04' 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:05:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:05:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:05:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:05:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:05:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:05:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:05:04' 2024-02-10T01:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:05:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:05:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:05:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:05:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:05:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap2 2024-02-10T01:05:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap2 2024-02-10T01:05:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2 2024-02-10T01:05:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:05:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:05:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:05:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:05:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:05:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:05:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:05:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:05:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:05:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:05:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:05:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:05:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:05:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:05:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:05:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:05:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:05:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:05:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:05:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:05:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:05:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:05:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:05:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 8 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:05:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:05:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:05:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=8 2024-02-10T01:05:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:05:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:05:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:05:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:05:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:05:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 8 = '' 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:05:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:05:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:05:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:05:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:05:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:05:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:05:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:05:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:05:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:05:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 8 = '' 2024-02-10T01:05:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:05:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:05:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:05:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:05:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:05:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:05:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:05:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:05:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:05:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 8 = '' 2024-02-10T01:05:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:05:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:05:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:05:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=8 2024-02-10T01:05:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 8 = 8 2024-02-10T01:05:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:05:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:05:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 4 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:05:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:05:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:05:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:05:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:05:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:05:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:05:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap1-clean ']' 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap1-clean 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1-clean 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:05:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:05:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:05:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap2 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:05:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:05:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:05:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap2-clean 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2-clean 2024-02-10T01:05:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:05:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:05:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:05:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:05:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=6 2024-02-10T01:05:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:05:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:05:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 87538688 bytes 2024-02-10T01:05:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 112325632 bytes 2024-02-10T01:05:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 139486208 bytes 2024-02-10T01:05:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 162350080 bytes 2024-02-10T01:05:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 177308672 bytes 2024-02-10T01:05:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 235223040 bytes 2024-02-10T01:05:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 300848128 bytes 2024-02-10T01:05:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 344423424 bytes 2024-02-10T01:05:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 419625984 bytes 2024-02-10T01:05:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 484445184 bytes 2024-02-10T01:05:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 513776640 bytes 2024-02-10T01:05:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 514408448 bytes 2024-02-10T01:05:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 564745216 bytes 2024-02-10T01:05:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 583087104 bytes 2024-02-10T01:05:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 679117824 bytes 2024-02-10T01:05:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 679726080 bytes 2024-02-10T01:05:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 717485056 bytes 2024-02-10T01:05:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 729316352 bytes 2024-02-10T01:05:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 781302784 bytes 2024-02-10T01:05:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 817483776 bytes 2024-02-10T01:05:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 863198208 bytes 2024-02-10T01:05:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 953598976 bytes 2024-02-10T01:05:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1005339648 bytes 2024-02-10T01:05:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1057310720 bytes 2024-02-10T01:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1082250240 bytes 2024-02-10T01:05:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1142436864 bytes 2024-02-10T01:05:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1143026688 bytes 2024-02-10T01:05:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1178335232 bytes 2024-02-10T01:05:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1225269248 bytes 2024-02-10T01:05:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1285805056 bytes 2024-02-10T01:05:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1344034816 bytes 2024-02-10T01:05:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1371803648 bytes 2024-02-10T01:05:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1488154624 bytes 2024-02-10T01:05:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1547659264 bytes 2024-02-10T01:05:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1574292480 bytes 2024-02-10T01:05:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1659006976 bytes 2024-02-10T01:05:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1694601216 bytes 2024-02-10T01:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1707658240 bytes 2024-02-10T01:05:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1800897536 bytes 2024-02-10T01:05:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1843902464 bytes 2024-02-10T01:05:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1885794304 bytes 2024-02-10T01:05:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1922814976 bytes 2024-02-10T01:05:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1933643776 bytes 2024-02-10T01:05:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1988727808 bytes 2024-02-10T01:05:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2056304640 bytes 2024-02-10T01:05:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2082581504 bytes 2024-02-10T01:05:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2118377472 bytes 2024-02-10T01:05:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2183146496 bytes 2024-02-10T01:05:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2279485440 bytes 2024-02-10T01:05:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2306373632 bytes 2024-02-10T01:05:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2338292736 bytes 2024-02-10T01:05:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2472184832 bytes 2024-02-10T01:05:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2511132672 bytes 2024-02-10T01:05:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2567612416 bytes 2024-02-10T01:05:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2627639296 bytes 2024-02-10T01:05:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:05:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:05:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:05:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:05:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:05:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:05:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:05:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:05:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:05:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:05:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:05:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:05:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:05:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527090,"remote_snapshot_timestamp":1707527090,"replay_state":"idle"} 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:05:34 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:05:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:05:34 2024-02-10T01:05:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:05:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527090,"remote_snapshot_timestamp":1707527090,"replay_state":"idle"} 2024-02-10T01:05:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:05:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:05:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:05:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:05:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:05:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:05:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:05:34' 2024-02-10T01:05:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:05:34' 2024-02-10T01:05:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:05:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:05:34' 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:05:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:05:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:05:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:05:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:05:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:05:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:05:34' 2024-02-10T01:05:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:05:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:05:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:05:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:05:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:05:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:05:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap3 2024-02-10T01:05:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap3 2024-02-10T01:05:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:05:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:05:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:05:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3 2024-02-10T01:05:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:06:18.468 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:06:18.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:06:18.469 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-02-10T01:06:18.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:06:18.469+0000 7f230a37d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 38619) UID: 1000 2024-02-10T01:06:25.470 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:06:25.471 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:06:25.471 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:06:25.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:06:25.471 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:06:25.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:06:25.475 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-02-10T01:06:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:06:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:06:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:06:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:06:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:06:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:06:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:06:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:06:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:06:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:06:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:06:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:06:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:06:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:06:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:06:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:06:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:06:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:06:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:06:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:06:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:06:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:06:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:06:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:06:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:06:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:06:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 1 2024-02-10T01:06:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:06:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:06:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:06:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:06:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:06:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:06:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:06:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:06:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:06:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:06:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:06:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:06:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:06:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:06:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:06:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:06:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 2 2024-02-10T01:06:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:06:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:06:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:06:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:06:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:06:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:06:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:06:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:06:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:06:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:06:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:06:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:06:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:06:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:06:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:06:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:06:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:06:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:06:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:06:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 4 2024-02-10T01:06:34.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.580126391803154) > weight (0.25) 2024-02-10T01:06:34.469 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-02-10T01:06:34.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-02-10T01:06:34.469 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-02-10T01:06:34.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi060.stderr:2024-02-10T01:06:34.468+0000 7f2256649640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38788) UID: 1000 2024-02-10T01:06:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:06:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:06:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:06:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:06:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:06:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:35.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:06:35.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:06:35.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:06:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:06:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:06:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:06:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:06:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:06:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:06:35.469 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-02-10T01:06:35.470 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:06:35.470 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-02-10T01:06:35.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-02-10T01:06:35.470 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-02-10T01:06:35.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-02-10T01:06:35.472 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-02-10T01:06:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:06:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:06:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:06:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:06:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 8 2024-02-10T01:06:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:06:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:06:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:06:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:06:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:06:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:06:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:06:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:06:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:06:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:06:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:06:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:06:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:06:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:06:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:06:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:06:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:06:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:06:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:06:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:06:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:06:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:06:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:06:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 8 2024-02-10T01:06:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:06:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:06:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:06:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:06:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:06:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:06:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:06:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:06:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:06:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:06:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:06:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:06:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:06:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:06:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:06:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:06:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:06:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:06:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:06:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:06:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:06:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:06:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:06:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:06:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:06:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:06:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:06:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:06:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:06:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:06:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:06:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:06:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 11 2024-02-10T01:06:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:06:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:06:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:06:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:06:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:06:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:06:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=11 2024-02-10T01:06:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:06:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:06:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:06:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:06:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:06:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:06:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:06:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:06:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:06:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:06:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:06:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 11 = '' 2024-02-10T01:06:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:06:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:06:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:06:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:06:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:06:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:06:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:06:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:06:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:06:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 11 = '' 2024-02-10T01:06:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:06:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:06:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:06:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:06:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:06:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:06:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:06:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:06:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:06:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:06:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 11 = '' 2024-02-10T01:06:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:06:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:06:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:06:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:06:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:06:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:06:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=11 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 11 = 11 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:06:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:06:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 6 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:06:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:05:46 2024 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:06:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap2-clean ']' 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap2-clean 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2-clean 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:06:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:06:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:06:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:06:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:06:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:06:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:06:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:06:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap3 2024-02-10T01:07:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3 2024-02-10T01:07:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:07:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:07:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:07:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:07:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:07:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap3-clean 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3-clean 2024-02-10T01:07:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:07:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:07:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:07:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:07:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=6 2024-02-10T01:07:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:07:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:07:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 42189824 bytes 2024-02-10T01:07:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 107060224 bytes 2024-02-10T01:07:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 159232000 bytes 2024-02-10T01:07:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 165891072 bytes 2024-02-10T01:07:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 201971712 bytes 2024-02-10T01:07:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 267146240 bytes 2024-02-10T01:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 284172288 bytes 2024-02-10T01:07:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 340084736 bytes 2024-02-10T01:07:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 341334016 bytes 2024-02-10T01:07:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 353645568 bytes 2024-02-10T01:07:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 412490752 bytes 2024-02-10T01:07:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 462104576 bytes 2024-02-10T01:07:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 521398272 bytes 2024-02-10T01:07:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 570995712 bytes 2024-02-10T01:07:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 649393152 bytes 2024-02-10T01:07:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 659397632 bytes 2024-02-10T01:07:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 749931520 bytes 2024-02-10T01:07:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 812581888 bytes 2024-02-10T01:07:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 864067584 bytes 2024-02-10T01:07:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 891511808 bytes 2024-02-10T01:07:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 920008704 bytes 2024-02-10T01:07:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 940052480 bytes 2024-02-10T01:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 940661760 bytes 2024-02-10T01:07:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 941258752 bytes 2024-02-10T01:07:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 967195648 bytes 2024-02-10T01:07:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 992712704 bytes 2024-02-10T01:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 993332224 bytes 2024-02-10T01:07:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1017865216 bytes 2024-02-10T01:07:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1051064320 bytes 2024-02-10T01:07:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1089598464 bytes 2024-02-10T01:07:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1155177472 bytes 2024-02-10T01:07:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1191442432 bytes 2024-02-10T01:07:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1228719104 bytes 2024-02-10T01:07:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1256607744 bytes 2024-02-10T01:07:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1298215936 bytes 2024-02-10T01:07:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1339143168 bytes 2024-02-10T01:07:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1363248128 bytes 2024-02-10T01:07:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1388775424 bytes 2024-02-10T01:07:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1435201536 bytes 2024-02-10T01:07:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1480058880 bytes 2024-02-10T01:07:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1492301824 bytes 2024-02-10T01:07:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1580311552 bytes 2024-02-10T01:07:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1593409536 bytes 2024-02-10T01:07:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1630220288 bytes 2024-02-10T01:07:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1663128576 bytes 2024-02-10T01:07:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1706665984 bytes 2024-02-10T01:07:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1749323776 bytes 2024-02-10T01:07:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1786916864 bytes 2024-02-10T01:07:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1846558720 bytes 2024-02-10T01:07:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1853347840 bytes 2024-02-10T01:07:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1882642432 bytes 2024-02-10T01:07:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1938009088 bytes 2024-02-10T01:07:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2006818816 bytes 2024-02-10T01:07:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2050654208 bytes 2024-02-10T01:07:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2102587392 bytes 2024-02-10T01:07:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2191462400 bytes 2024-02-10T01:07:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2211321856 bytes 2024-02-10T01:07:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2254016512 bytes 2024-02-10T01:07:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:07:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:07:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:07:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:07:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:07:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:07:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:07:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:07:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:07:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":47920593.45,"bytes_per_snapshot":527126528.0,"last_snapshot_bytes":527126528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527213,"remote_snapshot_timestamp":1707527213,"replay_state":"idle"} 2024-02-10T01:07:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.1 on smithi060 2024-02-10T01:07:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:07:04 2024-02-10T01:07:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:07:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:07:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:07:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:07:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:07:07 2024-02-10T01:07:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:07:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":47920593.45,"bytes_per_snapshot":527126528.0,"last_snapshot_bytes":527126528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527213,"remote_snapshot_timestamp":1707527213,"replay_state":"idle"} 2024-02-10T01:07:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:07:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:07:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:07:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:07:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:07:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:07:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:07:04' 2024-02-10T01:07:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:07:04' 2024-02-10T01:07:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:07:04' 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:07:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:07:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:07:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:07:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:07:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:07:04' 2024-02-10T01:07:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:07:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:07:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:07:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:07:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap4 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap4 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4 2024-02-10T01:07:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:07:24.477 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:07:24.476+0000 7fe71fd26640 -1 reset not still connected to 0x560a426d6340 2024-02-10T01:07:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:07:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:07:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:07:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:07:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:07:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:07:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:07:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:07:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:07:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:07:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:07:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:07:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:07:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:07:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:07:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:07:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:07:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:07:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:07:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:07:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:07:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:07:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:07:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:07:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:07:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:07:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:07:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:07:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:07:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:07:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:07:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:07:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:07:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:07:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:07:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:07:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:07:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:07:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:07:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:07:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:07:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:07:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:07:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:07:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:07:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:07:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:07:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:07:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:07:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:07:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:07:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:07:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 15 2024-02-10T01:07:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:07:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:07:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:07:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:07:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:07:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=15 2024-02-10T01:07:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:07:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:07:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:07:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:07:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:07:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:07:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:07:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:07:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:07:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:07:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 15 = '' 2024-02-10T01:07:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:07:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:07:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:07:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:07:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:07:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:07:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:07:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:07:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:07:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:07:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 15 = '' 2024-02-10T01:07:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:07:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:07:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:07:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:07:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:07:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:07:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:07:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:07:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 15 = '' 2024-02-10T01:07:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:07:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:07:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:07:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:07:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:07:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:07:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=15 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 15 = 15 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:07:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:07:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:07:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-02-10T01:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 8 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:06:59 2024 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap3-clean ']' 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap3-clean 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3-clean 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:07:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:07:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:07:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap4 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:07:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:07:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:07:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:07:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:07:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap4-clean 2024-02-10T01:07:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-02-10T01:07:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4-clean 2024-02-10T01:07:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:07:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:07:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:07:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:07:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:07:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:07:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:07:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=14 2024-02-10T01:07:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:07:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:07:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 6580224 bytes 2024-02-10T01:07:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 50314240 bytes 2024-02-10T01:07:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 81092608 bytes 2024-02-10T01:07:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 151697408 bytes 2024-02-10T01:07:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 169485312 bytes 2024-02-10T01:07:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 225867776 bytes 2024-02-10T01:07:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 245354496 bytes 2024-02-10T01:07:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 296743936 bytes 2024-02-10T01:07:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 346450944 bytes 2024-02-10T01:07:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 417249280 bytes 2024-02-10T01:07:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 510301184 bytes 2024-02-10T01:07:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 521769984 bytes 2024-02-10T01:07:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 527643648 bytes 2024-02-10T01:07:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 540182528 bytes 2024-02-10T01:07:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 657942528 bytes 2024-02-10T01:07:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 704337920 bytes 2024-02-10T01:07:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 729004032 bytes 2024-02-10T01:07:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 751923200 bytes 2024-02-10T01:07:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 866568192 bytes 2024-02-10T01:07:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 933932032 bytes 2024-02-10T01:07:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 974715904 bytes 2024-02-10T01:07:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1059334144 bytes 2024-02-10T01:07:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1072696320 bytes 2024-02-10T01:07:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1080675328 bytes 2024-02-10T01:07:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1119157248 bytes 2024-02-10T01:07:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1175050240 bytes 2024-02-10T01:07:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1195975680 bytes 2024-02-10T01:07:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1246366720 bytes 2024-02-10T01:07:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1270679552 bytes 2024-02-10T01:07:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1335902208 bytes 2024-02-10T01:07:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1425709056 bytes 2024-02-10T01:07:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1463636992 bytes 2024-02-10T01:07:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1476106240 bytes 2024-02-10T01:07:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1499602944 bytes 2024-02-10T01:07:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1542160384 bytes 2024-02-10T01:07:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1592211456 bytes 2024-02-10T01:07:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1662269440 bytes 2024-02-10T01:07:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1695518720 bytes 2024-02-10T01:07:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1711475712 bytes 2024-02-10T01:07:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1736745984 bytes 2024-02-10T01:07:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1755838464 bytes 2024-02-10T01:07:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1848171520 bytes 2024-02-10T01:07:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1892492288 bytes 2024-02-10T01:07:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1908101120 bytes 2024-02-10T01:07:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1947566080 bytes 2024-02-10T01:07:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2052718592 bytes 2024-02-10T01:07:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2118079488 bytes 2024-02-10T01:07:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2139098112 bytes 2024-02-10T01:07:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2159536128 bytes 2024-02-10T01:07:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2243310592 bytes 2024-02-10T01:07:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2307065856 bytes 2024-02-10T01:07:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2357792768 bytes 2024-02-10T01:07:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2375245824 bytes 2024-02-10T01:07:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2387863552 bytes 2024-02-10T01:07:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2447184896 bytes 2024-02-10T01:07:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2466910208 bytes 2024-02-10T01:07:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2471606272 bytes 2024-02-10T01:07:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2530449408 bytes 2024-02-10T01:07:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2631223296 bytes 2024-02-10T01:07:55.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7366294116075212) > weight (0.25) 2024-02-10T01:07:55.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.48697983078357654) > weight (0.25) 2024-02-10T01:07:55.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8715509904253972) > weight (0.25) 2024-02-10T01:07:55.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5265792602222363) > weight (0.25) 2024-02-10T01:07:55.474 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-02-10T01:07:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2710338560 bytes 2024-02-10T01:07:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2750040064 bytes 2024-02-10T01:07:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2750635008 bytes 2024-02-10T01:07:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2832183296 bytes 2024-02-10T01:07:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2850334720 bytes 2024-02-10T01:07:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2879581184 bytes 2024-02-10T01:07:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2917288960 bytes 2024-02-10T01:07:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3001538560 bytes 2024-02-10T01:07:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3086282752 bytes 2024-02-10T01:07:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3157029888 bytes 2024-02-10T01:07:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3183452160 bytes 2024-02-10T01:07:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3239065600 bytes 2024-02-10T01:07:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3301880832 bytes 2024-02-10T01:07:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3319413760 bytes 2024-02-10T01:07:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3332608000 bytes 2024-02-10T01:07:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3370084352 bytes 2024-02-10T01:07:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3497146368 bytes 2024-02-10T01:07:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3503713280 bytes 2024-02-10T01:07:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3525223424 bytes 2024-02-10T01:07:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3533637632 bytes 2024-02-10T01:07:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3547619328 bytes 2024-02-10T01:07:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3553111040 bytes 2024-02-10T01:07:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3615802368 bytes 2024-02-10T01:07:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3673523200 bytes 2024-02-10T01:07:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3694708736 bytes 2024-02-10T01:07:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3731663872 bytes 2024-02-10T01:07:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3777855488 bytes 2024-02-10T01:07:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3818395648 bytes 2024-02-10T01:07:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3915586560 bytes 2024-02-10T01:07:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3987449856 bytes 2024-02-10T01:07:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 4009556992 bytes 2024-02-10T01:07:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 4038838272 bytes 2024-02-10T01:07:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 4118833152 bytes 2024-02-10T01:07:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 4223736832 bytes 2024-02-10T01:07:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 4261906432 bytes 2024-02-10T01:07:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 4288610304 bytes 2024-02-10T01:07:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 4331482112 bytes 2024-02-10T01:07:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4347024384 bytes 2024-02-10T01:07:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4460431360 bytes 2024-02-10T01:07:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4486860800 bytes 2024-02-10T01:07:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4527964160 bytes 2024-02-10T01:07:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4565149696 bytes 2024-02-10T01:08:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 4605508608 bytes 2024-02-10T01:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 4664816640 bytes 2024-02-10T01:08:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 4715986944 bytes 2024-02-10T01:08:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10500 IOs, 4782704640 bytes 2024-02-10T01:08:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10600 IOs, 4824679424 bytes 2024-02-10T01:08:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10700 IOs, 4880034816 bytes 2024-02-10T01:08:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10800 IOs, 4935652352 bytes 2024-02-10T01:08:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10900 IOs, 4961468416 bytes 2024-02-10T01:08:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11000 IOs, 4979805184 bytes 2024-02-10T01:08:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11100 IOs, 5009958912 bytes 2024-02-10T01:08:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11200 IOs, 5063184384 bytes 2024-02-10T01:08:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11300 IOs, 5063799808 bytes 2024-02-10T01:08:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11400 IOs, 5085631488 bytes 2024-02-10T01:08:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11500 IOs, 5141506048 bytes 2024-02-10T01:08:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11600 IOs, 5213201408 bytes 2024-02-10T01:08:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11700 IOs, 5225232384 bytes 2024-02-10T01:08:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11800 IOs, 5259855872 bytes 2024-02-10T01:08:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11900 IOs, 5296486400 bytes 2024-02-10T01:08:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12000 IOs, 5323747328 bytes 2024-02-10T01:08:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12100 IOs, 5350002688 bytes 2024-02-10T01:08:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12200 IOs, 5396228096 bytes 2024-02-10T01:08:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12300 IOs, 5413262336 bytes 2024-02-10T01:08:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12400 IOs, 5447628800 bytes 2024-02-10T01:08:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12500 IOs, 5536704512 bytes 2024-02-10T01:08:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12600 IOs, 5570199552 bytes 2024-02-10T01:08:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12700 IOs, 5594181632 bytes 2024-02-10T01:08:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12800 IOs, 5658824704 bytes 2024-02-10T01:08:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:12900 IOs, 5720058880 bytes 2024-02-10T01:08:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:13000 IOs, 5794465792 bytes 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:08:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:08:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:08:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:08:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:08:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:08:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:08:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:08:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527126528.0,"last_snapshot_bytes":527126528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527213,"remote_snapshot_timestamp":1707527213,"replay_state":"idle"} 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.1 on smithi060 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:07:34 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:08:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:07:34 2024-02-10T01:08:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:08:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527126528.0,"last_snapshot_bytes":527126528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527213,"remote_snapshot_timestamp":1707527213,"replay_state":"idle"} 2024-02-10T01:08:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:08:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:08:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:08:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:08:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:08:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:07:34' 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:07:34' 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:07:34' 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:08:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:08:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:08:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:08:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:08:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:08:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:07:34' 2024-02-10T01:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:08:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:08:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:08:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:08:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:08:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap5 2024-02-10T01:08:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap5 2024-02-10T01:08:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5 2024-02-10T01:08:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:08:23.476 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8495592594241855) > weight (0.25) 2024-02-10T01:08:23.476 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5949069473600513) > weight (0.25) 2024-02-10T01:08:23.476 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-02-10T01:08:23.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-02-10T01:08:23.477 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-02-10T01:08:23.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:08:23.476+0000 7f4b0c730640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40693) UID: 1000 2024-02-10T01:08:30.478 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-02-10T01:08:30.478 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:08:30.478 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-02-10T01:08:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-02-10T01:08:30.479 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-02-10T01:08:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-02-10T01:08:30.483 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:08:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:08:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:08:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:08:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:08:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:08:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:08:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:08:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:08:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:08:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:08:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:08:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:08:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:08:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:08:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:08:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:08:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:08:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:08:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:08:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:08:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:08:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:08:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:08:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:08:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:08:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:08:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:08:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:08:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:08:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:08:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:08:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:08:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:08:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:08:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:08:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:08:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:08:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:08:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:08:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:08:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:08:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:08:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 18 2024-02-10T01:08:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:08:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:08:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:08:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:08:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:08:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=18 2024-02-10T01:08:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:08:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:08:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:08:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:08:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:08:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:08:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:08:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:08:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:08:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:08:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:08:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 18 = '' 2024-02-10T01:08:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:08:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:08:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:08:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:08:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:08:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:08:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:08:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:08:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:08:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:08:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 18 = '' 2024-02-10T01:08:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:08:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:08:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:08:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:08:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:08:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:08:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:08:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:08:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:08:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 18 = '' 2024-02-10T01:08:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:08:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:08:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:08:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=18 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 18 = 18 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:08:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:08:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-02-10T01:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 10 2024-02-10T01:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:06:59 2024 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:08:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:08:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap4-clean ']' 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap4-clean 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4-clean 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:08:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:08:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:08:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap5 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:08:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:08:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:08:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:08:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap5-clean 2024-02-10T01:08:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-02-10T01:08:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5-clean 2024-02-10T01:08:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:08:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:08:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:08:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:08:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:08:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:08:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:08:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 18455552 bytes 2024-02-10T01:08:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 52563968 bytes 2024-02-10T01:08:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 72476672 bytes 2024-02-10T01:08:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 112604160 bytes 2024-02-10T01:08:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 138647552 bytes 2024-02-10T01:08:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 207975424 bytes 2024-02-10T01:08:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 225291264 bytes 2024-02-10T01:08:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 260056064 bytes 2024-02-10T01:08:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 316413952 bytes 2024-02-10T01:08:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 340848640 bytes 2024-02-10T01:08:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 395485184 bytes 2024-02-10T01:08:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 439044096 bytes 2024-02-10T01:08:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 458671104 bytes 2024-02-10T01:08:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 468362240 bytes 2024-02-10T01:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 525601792 bytes 2024-02-10T01:08:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 601997312 bytes 2024-02-10T01:08:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 612441088 bytes 2024-02-10T01:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 651513856 bytes 2024-02-10T01:08:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 731150336 bytes 2024-02-10T01:08:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 792010752 bytes 2024-02-10T01:08:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 842796032 bytes 2024-02-10T01:08:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 864401408 bytes 2024-02-10T01:08:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 920828928 bytes 2024-02-10T01:08:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 925139968 bytes 2024-02-10T01:08:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 956105728 bytes 2024-02-10T01:08:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1002618880 bytes 2024-02-10T01:08:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1046750208 bytes 2024-02-10T01:08:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1048259584 bytes 2024-02-10T01:08:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1072832512 bytes 2024-02-10T01:08:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1089367040 bytes 2024-02-10T01:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1135160320 bytes 2024-02-10T01:08:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1167763456 bytes 2024-02-10T01:08:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1210131456 bytes 2024-02-10T01:08:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1266155520 bytes 2024-02-10T01:08:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1359063040 bytes 2024-02-10T01:08:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1371681792 bytes 2024-02-10T01:08:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1381509120 bytes 2024-02-10T01:08:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1388309504 bytes 2024-02-10T01:08:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1416591360 bytes 2024-02-10T01:08:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1442007040 bytes 2024-02-10T01:08:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1504936960 bytes 2024-02-10T01:08:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1554555904 bytes 2024-02-10T01:08:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1589837824 bytes 2024-02-10T01:08:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1625987072 bytes 2024-02-10T01:08:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1644320768 bytes 2024-02-10T01:08:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1655497728 bytes 2024-02-10T01:08:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1682180096 bytes 2024-02-10T01:08:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1746105344 bytes 2024-02-10T01:08:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1760822272 bytes 2024-02-10T01:08:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1843437568 bytes 2024-02-10T01:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1921760256 bytes 2024-02-10T01:08:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1953978368 bytes 2024-02-10T01:08:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 1957514240 bytes 2024-02-10T01:08:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 1978670080 bytes 2024-02-10T01:08:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2028354560 bytes 2024-02-10T01:08:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2054486016 bytes 2024-02-10T01:08:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2076136448 bytes 2024-02-10T01:08:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2136130560 bytes 2024-02-10T01:08:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2142157824 bytes 2024-02-10T01:08:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2164943872 bytes 2024-02-10T01:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2179214336 bytes 2024-02-10T01:08:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2228975616 bytes 2024-02-10T01:08:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2259034112 bytes 2024-02-10T01:08:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2306368512 bytes 2024-02-10T01:08:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2363683840 bytes 2024-02-10T01:08:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2401142784 bytes 2024-02-10T01:08:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2456744960 bytes 2024-02-10T01:08:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2508155904 bytes 2024-02-10T01:08:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2515869696 bytes 2024-02-10T01:08:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2552941568 bytes 2024-02-10T01:08:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2609696768 bytes 2024-02-10T01:08:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2610319360 bytes 2024-02-10T01:08:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2679352320 bytes 2024-02-10T01:08:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 2679954432 bytes 2024-02-10T01:08:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 2702636032 bytes 2024-02-10T01:08:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 2727226368 bytes 2024-02-10T01:08:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 2753838080 bytes 2024-02-10T01:08:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 2781381632 bytes 2024-02-10T01:08:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 2832456704 bytes 2024-02-10T01:08:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 2853566464 bytes 2024-02-10T01:08:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 2906323968 bytes 2024-02-10T01:08:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 2917415936 bytes 2024-02-10T01:08:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 2943004672 bytes 2024-02-10T01:08:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3016599552 bytes 2024-02-10T01:08:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3075880960 bytes 2024-02-10T01:08:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3116158976 bytes 2024-02-10T01:08:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3136613376 bytes 2024-02-10T01:08:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3172182016 bytes 2024-02-10T01:08:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3203770368 bytes 2024-02-10T01:08:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3266780160 bytes 2024-02-10T01:08:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3302913024 bytes 2024-02-10T01:08:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3303539712 bytes 2024-02-10T01:08:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3336175616 bytes 2024-02-10T01:08:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3381779456 bytes 2024-02-10T01:08:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3399124992 bytes 2024-02-10T01:08:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3431329792 bytes 2024-02-10T01:08:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 3471234048 bytes 2024-02-10T01:08:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 3519466496 bytes 2024-02-10T01:08:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 3544851456 bytes 2024-02-10T01:08:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 3581230080 bytes 2024-02-10T01:08:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 3589693440 bytes 2024-02-10T01:08:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 3630942208 bytes 2024-02-10T01:08:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 3674350592 bytes 2024-02-10T01:08:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 3674952704 bytes 2024-02-10T01:08:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10500 IOs, 3721196544 bytes 2024-02-10T01:08:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10600 IOs, 3803205632 bytes 2024-02-10T01:08:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10700 IOs, 3824655360 bytes 2024-02-10T01:08:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10800 IOs, 3857587200 bytes 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:08:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:08:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:08:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:08:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:08:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:08:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:08:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:08:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:08:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:08:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:08:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:08:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527259,"remote_snapshot_timestamp":1707527259,"replay_state":"idle"} 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.1 on smithi060 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:08:34 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:08:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:08:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:08:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:08:34 2024-02-10T01:08:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:08:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527259,"remote_snapshot_timestamp":1707527259,"replay_state":"idle"} 2024-02-10T01:08:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:08:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:08:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:08:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:08:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:08:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:08:34' 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:08:34' 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:08:34' 2024-02-10T01:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:08:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:08:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:08:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:08:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:08:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:08:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:08:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:08:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:08:34' 2024-02-10T01:08:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:08:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:08:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:08:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:08:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap6 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap6 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6 2024-02-10T01:08:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:09:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:09:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:09:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:09:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:09:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:09:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:09:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:09:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:09:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:09:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:09:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:09:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:09:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:09:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:09:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:09:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:09:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:09:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:09:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:09:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:09:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:09:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:09:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:09:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:09:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:09:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:09:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:09:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:09:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:09:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:09:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:09:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:09:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:09:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:09:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:09:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:09:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:09:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:09:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:09:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:09:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:09:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:09:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:09:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:09:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:09:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:09:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:09:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:09:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:09:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:09:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:09:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:09:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:09:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:09:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 21 2024-02-10T01:09:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:09:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:09:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:09:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:09:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:09:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=21 2024-02-10T01:09:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:09:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:09:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:09:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:09:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:09:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:09:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:09:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:09:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 21 = '' 2024-02-10T01:09:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:09:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:09:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:09:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:09:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:09:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:09:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:09:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:09:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:09:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:09:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 21 = '' 2024-02-10T01:09:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:09:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:09:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:09:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:09:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:09:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:09:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:09:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:09:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:09:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:09:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 21 = '' 2024-02-10T01:09:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:09:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:09:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:09:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:09:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=21 2024-02-10T01:09:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 21 = 21 2024-02-10T01:09:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:09:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:09:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 12 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:09:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:08:41 2024 2024-02-10T01:09:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:09:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:09:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:09:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:09:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:09:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap5-clean ']' 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap5-clean 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5-clean 2024-02-10T01:09:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:09:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:09:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:09:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:09:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:09:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap6 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:09:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:09:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:09:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap6-clean 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6-clean 2024-02-10T01:09:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:09:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:09:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:09:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:09:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=12 2024-02-10T01:09:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:09:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:09:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 10060800 bytes 2024-02-10T01:09:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 43039744 bytes 2024-02-10T01:09:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 86940672 bytes 2024-02-10T01:09:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 110595072 bytes 2024-02-10T01:09:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 116382720 bytes 2024-02-10T01:09:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 234891264 bytes 2024-02-10T01:09:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 266867712 bytes 2024-02-10T01:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 334664704 bytes 2024-02-10T01:09:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 366988288 bytes 2024-02-10T01:09:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 379392000 bytes 2024-02-10T01:09:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 433934336 bytes 2024-02-10T01:09:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 527866880 bytes 2024-02-10T01:09:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 559575040 bytes 2024-02-10T01:09:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 614848512 bytes 2024-02-10T01:09:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 693860352 bytes 2024-02-10T01:09:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 739279872 bytes 2024-02-10T01:09:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 758848512 bytes 2024-02-10T01:09:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 793861120 bytes 2024-02-10T01:09:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 851641344 bytes 2024-02-10T01:09:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 907702272 bytes 2024-02-10T01:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 990536704 bytes 2024-02-10T01:09:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1006639104 bytes 2024-02-10T01:09:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1062185984 bytes 2024-02-10T01:09:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1137017856 bytes 2024-02-10T01:09:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1178961920 bytes 2024-02-10T01:09:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1217370112 bytes 2024-02-10T01:09:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1250874368 bytes 2024-02-10T01:09:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1284602880 bytes 2024-02-10T01:09:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1285221376 bytes 2024-02-10T01:09:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1341061120 bytes 2024-02-10T01:09:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1405854720 bytes 2024-02-10T01:09:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1433986048 bytes 2024-02-10T01:09:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1468802048 bytes 2024-02-10T01:09:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1490336768 bytes 2024-02-10T01:09:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1587714048 bytes 2024-02-10T01:09:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1611240448 bytes 2024-02-10T01:09:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1611851776 bytes 2024-02-10T01:09:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1646425088 bytes 2024-02-10T01:09:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1661119488 bytes 2024-02-10T01:09:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1661749248 bytes 2024-02-10T01:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1670616064 bytes 2024-02-10T01:09:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1696626688 bytes 2024-02-10T01:09:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1738702848 bytes 2024-02-10T01:09:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1796202496 bytes 2024-02-10T01:09:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1821648896 bytes 2024-02-10T01:09:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1887565824 bytes 2024-02-10T01:09:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1893487616 bytes 2024-02-10T01:09:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1965164544 bytes 2024-02-10T01:09:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2012791808 bytes 2024-02-10T01:09:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2031503360 bytes 2024-02-10T01:09:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2071609344 bytes 2024-02-10T01:09:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2110068736 bytes 2024-02-10T01:09:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2130442240 bytes 2024-02-10T01:09:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2171855872 bytes 2024-02-10T01:09:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2210962432 bytes 2024-02-10T01:09:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2283158528 bytes 2024-02-10T01:09:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2293079040 bytes 2024-02-10T01:09:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2314265600 bytes 2024-02-10T01:09:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2339519488 bytes 2024-02-10T01:09:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2392454144 bytes 2024-02-10T01:09:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2431730688 bytes 2024-02-10T01:09:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2502686720 bytes 2024-02-10T01:09:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2503293952 bytes 2024-02-10T01:09:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2560321536 bytes 2024-02-10T01:09:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2634273792 bytes 2024-02-10T01:09:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2655799296 bytes 2024-02-10T01:09:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2685663232 bytes 2024-02-10T01:09:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2797251584 bytes 2024-02-10T01:09:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2823871488 bytes 2024-02-10T01:09:45.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2898282496 bytes 2024-02-10T01:09:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2981707776 bytes 2024-02-10T01:09:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2984804352 bytes 2024-02-10T01:09:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3042129920 bytes 2024-02-10T01:09:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3060371456 bytes 2024-02-10T01:09:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3141224448 bytes 2024-02-10T01:09:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3184868352 bytes 2024-02-10T01:09:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3224318976 bytes 2024-02-10T01:09:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3286963200 bytes 2024-02-10T01:09:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3331553280 bytes 2024-02-10T01:09:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3344870400 bytes 2024-02-10T01:09:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3414961152 bytes 2024-02-10T01:09:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3452012544 bytes 2024-02-10T01:09:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3485878272 bytes 2024-02-10T01:09:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3533586432 bytes 2024-02-10T01:09:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3591946240 bytes 2024-02-10T01:09:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3634493440 bytes 2024-02-10T01:09:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3682209792 bytes 2024-02-10T01:09:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3722483712 bytes 2024-02-10T01:09:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3825924096 bytes 2024-02-10T01:09:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3874386944 bytes 2024-02-10T01:09:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3929201664 bytes 2024-02-10T01:09:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3934045184 bytes 2024-02-10T01:09:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3991672832 bytes 2024-02-10T01:09:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 4077435904 bytes 2024-02-10T01:09:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 4186019840 bytes 2024-02-10T01:09:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 4206283776 bytes 2024-02-10T01:09:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4309234688 bytes 2024-02-10T01:09:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4322057216 bytes 2024-02-10T01:09:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4351416320 bytes 2024-02-10T01:09:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4389972992 bytes 2024-02-10T01:09:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4428762112 bytes 2024-02-10T01:09:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:09:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:09:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:09:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:09:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:09:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:09:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3075822.93,"bytes_per_snapshot":536277504.0,"last_snapshot_bytes":536475648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527367,"remote_snapshot_timestamp":1707527367,"replay_state":"idle"} 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.1 on smithi060 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:09:34 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:09:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:09:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:09:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:09:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:09:34 2024-02-10T01:09:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:09:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3075822.93,"bytes_per_snapshot":536277504.0,"last_snapshot_bytes":536475648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527367,"remote_snapshot_timestamp":1707527367,"replay_state":"idle"} 2024-02-10T01:09:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:09:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:09:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:09:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:09:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:09:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:09:34' 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:09:34' 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:09:34' 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:09:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:09:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:09:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:09:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:09:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:09:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:09:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:09:34' 2024-02-10T01:09:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:09:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:09:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:09:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:09:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:09:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap7 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap7 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7 2024-02-10T01:09:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:09:50.475 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9384588049755327) > weight (0.25) 2024-02-10T01:09:50.475 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8032227933122864) > weight (0.25) 2024-02-10T01:09:50.476 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-02-10T01:09:50.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-02-10T01:09:50.476 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-02-10T01:09:50.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi060.stderr:2024-02-10T01:09:50.476+0000 7fd1db7cb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 38836) UID: 1000 2024-02-10T01:09:54.477 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-02-10T01:09:54.477 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:09:54.477 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-02-10T01:09:54.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-02-10T01:09:54.478 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-02-10T01:09:54.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-02-10T01:09:54.481 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-02-10T01:10:17.484 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:10:17.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:10:17.485 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-02-10T01:10:17.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:10:17.484+0000 7fbbab8df640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41424) UID: 1000 2024-02-10T01:10:18.485 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:10:18.486 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:10:18.486 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:10:18.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:10:18.486 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:10:18.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:10:18.490 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-02-10T01:10:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:10:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:10:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:10:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:10:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:10:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:10:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:10:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:10:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:10:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:10:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:10:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:10:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:10:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:10:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:10:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:10:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:10:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:10:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:10:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:10:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:10:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:10:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:10:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:10:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:10:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:10:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:10:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:10:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:10:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:10:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:10:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:10:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:10:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:10:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:10:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:10:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:10:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:10:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:10:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:10:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:10:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:10:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:10:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 24 2024-02-10T01:10:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:10:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:10:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:10:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:10:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:10:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=24 2024-02-10T01:10:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:10:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:10:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:10:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:10:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:10:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:10:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:10:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:10:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:10:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:10:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 24 = '' 2024-02-10T01:10:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:10:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:10:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:10:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:10:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:10:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:10:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:10:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:10:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:10:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:10:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 24 = '' 2024-02-10T01:10:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:10:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:10:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:10:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:10:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:10:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:10:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:10:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:10:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:10:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 24 = '' 2024-02-10T01:10:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:10:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:10:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:10:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:10:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:10:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=24 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 24 = 24 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:10:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:10:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-02-10T01:10:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:10:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:10:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 14 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:08:41 2024 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:10:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:10:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap6-clean ']' 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap6-clean 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6-clean 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:10:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:10:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:10:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:10:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:10:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap7 2024-02-10T01:10:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7 2024-02-10T01:10:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:10:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:10:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:10:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:10:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:10:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:10:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap7-clean 2024-02-10T01:10:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-02-10T01:10:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7-clean 2024-02-10T01:10:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:10:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:10:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:10:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:10:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:10:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:10:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:10:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 19077120 bytes 2024-02-10T01:10:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 32202752 bytes 2024-02-10T01:10:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 108767232 bytes 2024-02-10T01:10:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 135156736 bytes 2024-02-10T01:10:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 236247040 bytes 2024-02-10T01:10:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 280444928 bytes 2024-02-10T01:10:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 392462336 bytes 2024-02-10T01:10:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 422496256 bytes 2024-02-10T01:10:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 459634688 bytes 2024-02-10T01:10:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 492258304 bytes 2024-02-10T01:10:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 520883200 bytes 2024-02-10T01:10:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 591577088 bytes 2024-02-10T01:10:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 683887616 bytes 2024-02-10T01:10:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 697621504 bytes 2024-02-10T01:10:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 769145856 bytes 2024-02-10T01:10:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 788718592 bytes 2024-02-10T01:10:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 833689600 bytes 2024-02-10T01:10:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 866979840 bytes 2024-02-10T01:10:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 951232512 bytes 2024-02-10T01:10:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1004848128 bytes 2024-02-10T01:10:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1025451008 bytes 2024-02-10T01:10:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1059249152 bytes 2024-02-10T01:10:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1184601088 bytes 2024-02-10T01:10:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1203468288 bytes 2024-02-10T01:10:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1244014592 bytes 2024-02-10T01:10:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1306515456 bytes 2024-02-10T01:10:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1339815936 bytes 2024-02-10T01:10:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1361639424 bytes 2024-02-10T01:10:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1382931456 bytes 2024-02-10T01:10:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1431900160 bytes 2024-02-10T01:10:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1518930944 bytes 2024-02-10T01:10:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1564587008 bytes 2024-02-10T01:10:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1592646656 bytes 2024-02-10T01:10:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1670313984 bytes 2024-02-10T01:10:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1730299904 bytes 2024-02-10T01:10:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1815139328 bytes 2024-02-10T01:10:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1815736320 bytes 2024-02-10T01:10:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1859532800 bytes 2024-02-10T01:10:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1894757376 bytes 2024-02-10T01:10:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1916560384 bytes 2024-02-10T01:10:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1936281600 bytes 2024-02-10T01:10:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 2044386304 bytes 2024-02-10T01:10:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2150656000 bytes 2024-02-10T01:10:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2237231104 bytes 2024-02-10T01:10:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2269889536 bytes 2024-02-10T01:10:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2322849792 bytes 2024-02-10T01:10:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2346231808 bytes 2024-02-10T01:10:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2395133952 bytes 2024-02-10T01:10:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2459583488 bytes 2024-02-10T01:10:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2520996864 bytes 2024-02-10T01:10:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2571470848 bytes 2024-02-10T01:10:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2642122752 bytes 2024-02-10T01:10:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2694693888 bytes 2024-02-10T01:10:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2732623872 bytes 2024-02-10T01:10:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2734945280 bytes 2024-02-10T01:10:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:10:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:10:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:10:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:10:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:10:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:10:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:10:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:10:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:10:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:10:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:10:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:10:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:10:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":44355242.67,"bytes_per_snapshot":532262912.0,"last_snapshot_bytes":532262912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527421,"remote_snapshot_timestamp":1707527421,"replay_state":"idle"} 2024-02-10T01:10:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.1 on smithi060 2024-02-10T01:10:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:10:34 2024-02-10T01:10:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:10:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:10:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:10:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:10:34 2024-02-10T01:10:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:10:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":44355242.67,"bytes_per_snapshot":532262912.0,"last_snapshot_bytes":532262912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527421,"remote_snapshot_timestamp":1707527421,"replay_state":"idle"} 2024-02-10T01:10:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:10:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:10:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:10:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:10:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:10:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:10:34' 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:10:34' 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:10:34' 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:10:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:10:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:10:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:10:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:10:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:10:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:10:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:10:34' 2024-02-10T01:10:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:10:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:10:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:10:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:10:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:10:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:10:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap8 2024-02-10T01:10:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap8 2024-02-10T01:10:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:10:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:10:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:10:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8 2024-02-10T01:10:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:11:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:11:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:11:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:11:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:11:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:11:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:11:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:11:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:11:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:11:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:11:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:11:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:11:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:11:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:11:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:11:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:11:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:11:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:11:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:11:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:11:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:11:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:11:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:11:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:11:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:11:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:11:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:11:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:11:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:11:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:11:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:11:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:11:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:11:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:11:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:11:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:11:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:11:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 27 2024-02-10T01:11:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:11:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:11:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:11:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:11:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:11:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=27 2024-02-10T01:11:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:11:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:11:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:11:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:11:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:11:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 27 = '' 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:11:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:11:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:11:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:11:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:11:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:11:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 27 = '' 2024-02-10T01:11:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:11:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:11:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:11:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:11:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:11:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:11:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:11:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:11:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:11:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:11:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 27 = '' 2024-02-10T01:11:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:11:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:11:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:11:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:11:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:11:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:11:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=27 2024-02-10T01:11:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 27 = 27 2024-02-10T01:11:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:11:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:11:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-02-10T01:11:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:11:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:11:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:11:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 16 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:10:27 2024 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:11:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap7-clean ']' 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap7-clean 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7-clean 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:11:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:11:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:11:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:11:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap8 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:11:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:11:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:11:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap8-clean 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8-clean 2024-02-10T01:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:11:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:11:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:11:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:11:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:11:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:11:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:11:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 28776448 bytes 2024-02-10T01:11:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 98976768 bytes 2024-02-10T01:11:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 139563008 bytes 2024-02-10T01:11:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 186589184 bytes 2024-02-10T01:11:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 214990848 bytes 2024-02-10T01:11:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 251301888 bytes 2024-02-10T01:11:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 251930624 bytes 2024-02-10T01:11:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 315888640 bytes 2024-02-10T01:11:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 353553408 bytes 2024-02-10T01:11:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 419798016 bytes 2024-02-10T01:11:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 440396800 bytes 2024-02-10T01:11:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 461061120 bytes 2024-02-10T01:11:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 555427840 bytes 2024-02-10T01:11:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 563081216 bytes 2024-02-10T01:11:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 645086208 bytes 2024-02-10T01:11:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 676024320 bytes 2024-02-10T01:11:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 712810496 bytes 2024-02-10T01:11:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 759948288 bytes 2024-02-10T01:11:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 783084544 bytes 2024-02-10T01:11:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 845622272 bytes 2024-02-10T01:11:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 920453120 bytes 2024-02-10T01:11:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 953821184 bytes 2024-02-10T01:11:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 971372544 bytes 2024-02-10T01:11:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1006277632 bytes 2024-02-10T01:11:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1042385920 bytes 2024-02-10T01:11:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1061307392 bytes 2024-02-10T01:11:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1099319296 bytes 2024-02-10T01:11:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1165788160 bytes 2024-02-10T01:11:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1218082816 bytes 2024-02-10T01:11:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1235189760 bytes 2024-02-10T01:11:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1260778496 bytes 2024-02-10T01:11:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1350306816 bytes 2024-02-10T01:11:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1398109184 bytes 2024-02-10T01:11:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1402906624 bytes 2024-02-10T01:11:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1408574464 bytes 2024-02-10T01:11:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1495868416 bytes 2024-02-10T01:11:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1543427072 bytes 2024-02-10T01:11:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1602666496 bytes 2024-02-10T01:11:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1638672384 bytes 2024-02-10T01:11:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1659497472 bytes 2024-02-10T01:11:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1676314624 bytes 2024-02-10T01:11:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1700097024 bytes 2024-02-10T01:11:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1735862272 bytes 2024-02-10T01:11:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1785848832 bytes 2024-02-10T01:11:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1838254080 bytes 2024-02-10T01:11:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1898589184 bytes 2024-02-10T01:11:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1918077952 bytes 2024-02-10T01:11:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1942639616 bytes 2024-02-10T01:11:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1982003200 bytes 2024-02-10T01:11:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1998617600 bytes 2024-02-10T01:11:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2053624832 bytes 2024-02-10T01:11:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2142800896 bytes 2024-02-10T01:11:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2163008512 bytes 2024-02-10T01:11:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2217664512 bytes 2024-02-10T01:11:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2263446528 bytes 2024-02-10T01:11:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2285113344 bytes 2024-02-10T01:11:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2336513024 bytes 2024-02-10T01:11:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2346488832 bytes 2024-02-10T01:11:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2368855040 bytes 2024-02-10T01:11:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2403236864 bytes 2024-02-10T01:11:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2412410880 bytes 2024-02-10T01:11:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2507019264 bytes 2024-02-10T01:11:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2588548096 bytes 2024-02-10T01:11:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2612915200 bytes 2024-02-10T01:11:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2682873856 bytes 2024-02-10T01:11:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2710680576 bytes 2024-02-10T01:11:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2711315456 bytes 2024-02-10T01:11:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2753443840 bytes 2024-02-10T01:11:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2761513984 bytes 2024-02-10T01:11:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2829496320 bytes 2024-02-10T01:11:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2929020928 bytes 2024-02-10T01:11:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3003871232 bytes 2024-02-10T01:11:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3014412288 bytes 2024-02-10T01:11:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3075703808 bytes 2024-02-10T01:11:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3104688128 bytes 2024-02-10T01:11:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3156322304 bytes 2024-02-10T01:11:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3208906752 bytes 2024-02-10T01:11:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3232601088 bytes 2024-02-10T01:11:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3277462528 bytes 2024-02-10T01:11:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3333804032 bytes 2024-02-10T01:11:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3409843200 bytes 2024-02-10T01:11:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3493164032 bytes 2024-02-10T01:11:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3578881024 bytes 2024-02-10T01:11:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3662488576 bytes 2024-02-10T01:11:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3690739712 bytes 2024-02-10T01:11:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3745984512 bytes 2024-02-10T01:11:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3819101184 bytes 2024-02-10T01:11:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3861046272 bytes 2024-02-10T01:11:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3869136896 bytes 2024-02-10T01:11:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3939922944 bytes 2024-02-10T01:11:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 4005306368 bytes 2024-02-10T01:11:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 4037684224 bytes 2024-02-10T01:11:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 4061644800 bytes 2024-02-10T01:11:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 4080327680 bytes 2024-02-10T01:11:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 4144809984 bytes 2024-02-10T01:11:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 4161812480 bytes 2024-02-10T01:11:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4212460544 bytes 2024-02-10T01:11:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4267431936 bytes 2024-02-10T01:11:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4292515840 bytes 2024-02-10T01:11:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4305223680 bytes 2024-02-10T01:11:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4321262592 bytes 2024-02-10T01:11:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 4395597824 bytes 2024-02-10T01:11:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 4431035392 bytes 2024-02-10T01:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 4455260160 bytes 2024-02-10T01:11:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10500 IOs, 4484712448 bytes 2024-02-10T01:11:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10600 IOs, 4594050048 bytes 2024-02-10T01:11:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10700 IOs, 4640814080 bytes 2024-02-10T01:11:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10800 IOs, 4697943040 bytes 2024-02-10T01:11:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10900 IOs, 4721765376 bytes 2024-02-10T01:11:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:11000 IOs, 4778300416 bytes 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:11:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:11:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:11:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:11:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:11:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:11:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:11:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:11:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:11:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:11:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532262912.0,"last_snapshot_bytes":532262912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527421,"remote_snapshot_timestamp":1707527421,"replay_state":"idle"} 2024-02-10T01:11:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.1 on smithi060 2024-02-10T01:11:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:11:04 2024-02-10T01:11:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:11:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:11:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:11:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:11:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:11:04 2024-02-10T01:11:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:11:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532262912.0,"last_snapshot_bytes":532262912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527421,"remote_snapshot_timestamp":1707527421,"replay_state":"idle"} 2024-02-10T01:11:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:11:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:11:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:11:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:11:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:11:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:11:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:11:04' 2024-02-10T01:11:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:11:04' 2024-02-10T01:11:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:11:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:11:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:11:04' 2024-02-10T01:11:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:11:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:11:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:11:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:11:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:11:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:11:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:11:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:11:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:11:04' 2024-02-10T01:11:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:11:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:11:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:11:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:11:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:11:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:11:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap9 2024-02-10T01:11:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap9 2024-02-10T01:11:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:11:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:11:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:11:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9 2024-02-10T01:11:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:11:47.482 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9382087770124744) > weight (0.25) 2024-02-10T01:11:47.483 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7771523124080484) > weight (0.25) 2024-02-10T01:11:47.483 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-02-10T01:11:47.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-02-10T01:11:47.483 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-02-10T01:11:47.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi060.stderr:2024-02-10T01:11:47.484+0000 7f3671878640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 43547) UID: 1000 2024-02-10T01:11:48.484 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-02-10T01:11:48.485 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:11:48.485 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-02-10T01:11:48.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-02-10T01:11:48.485 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-02-10T01:11:48.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-02-10T01:11:48.488 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-02-10T01:12:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:12:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:12:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:12:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:12:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:12:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:12:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:12:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:12:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:12:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:12:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:12:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:12:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:12:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:12:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:12:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:12:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:12:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:12:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:12:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:12:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:12:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:12:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:12:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:12:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:12:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:12:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:12:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:12:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:12:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:12:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:12:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:12:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:12:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:12:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:12:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:12:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:12:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:12:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:12:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:12:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:12:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:12:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:12:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 30 2024-02-10T01:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:12:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:12:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:12:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=30 2024-02-10T01:12:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:12:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 30 = '' 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:12:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:12:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:12:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:12:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:12:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 30 = '' 2024-02-10T01:12:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:12:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:12:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 30 = '' 2024-02-10T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=30 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 30 = 30 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9 2024-02-10T01:12:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:12:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:12:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:12:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-02-10T01:12:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:12:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 18 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:10:27 2024 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:12:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:12:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap8-clean ']' 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap8-clean 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8-clean 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:12:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:12:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:12:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:12:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap9 2024-02-10T01:12:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9 2024-02-10T01:12:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:12:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:12:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:12:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:12:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:12:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:12:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap9-clean 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9-clean 2024-02-10T01:12:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:12:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:12:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror test 2024-02-10T01:12:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-02-10T01:12:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 587776 bytes 2024-02-10T01:12:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 73018368 bytes 2024-02-10T01:12:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 125593600 bytes 2024-02-10T01:12:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 207370240 bytes 2024-02-10T01:12:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 290555904 bytes 2024-02-10T01:12:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 318536704 bytes 2024-02-10T01:12:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 393428992 bytes 2024-02-10T01:12:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 453655552 bytes 2024-02-10T01:12:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 529876992 bytes 2024-02-10T01:12:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 564506624 bytes 2024-02-10T01:12:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 663342080 bytes 2024-02-10T01:12:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 709621760 bytes 2024-02-10T01:12:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 822366208 bytes 2024-02-10T01:12:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 871920640 bytes 2024-02-10T01:12:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 969675776 bytes 2024-02-10T01:12:13.491 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6626046193179805) > weight (0.25) 2024-02-10T01:12:13.491 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-02-10T01:12:13.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-02-10T01:12:13.492 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-02-10T01:12:13.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi060.stderr:2024-02-10T01:12:13.491+0000 7f56c0329640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 38636) UID: 1000 2024-02-10T01:12:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 979454976 bytes 2024-02-10T01:12:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 1041644544 bytes 2024-02-10T01:12:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 1178931200 bytes 2024-02-10T01:12:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 1266017280 bytes 2024-02-10T01:12:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1370549248 bytes 2024-02-10T01:12:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1404753920 bytes 2024-02-10T01:12:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1456566272 bytes 2024-02-10T01:12:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1524957184 bytes 2024-02-10T01:12:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1569703936 bytes 2024-02-10T01:12:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1644261376 bytes 2024-02-10T01:12:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1660624896 bytes 2024-02-10T01:12:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1715834880 bytes 2024-02-10T01:12:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1737449472 bytes 2024-02-10T01:12:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1792194560 bytes 2024-02-10T01:12:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1792820224 bytes 2024-02-10T01:12:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1801340928 bytes 2024-02-10T01:12:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1860669440 bytes 2024-02-10T01:12:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1902531584 bytes 2024-02-10T01:12:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1930072064 bytes 2024-02-10T01:12:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1976660992 bytes 2024-02-10T01:12:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 2047588352 bytes 2024-02-10T01:12:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 2092212224 bytes 2024-02-10T01:12:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 2124210176 bytes 2024-02-10T01:12:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 2133745664 bytes 2024-02-10T01:12:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 2153086976 bytes 2024-02-10T01:12:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 2156308480 bytes 2024-02-10T01:12:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 2202933248 bytes 2024-02-10T01:12:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2233619456 bytes 2024-02-10T01:12:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2239850496 bytes 2024-02-10T01:12:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2240872448 bytes 2024-02-10T01:12:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2250887168 bytes 2024-02-10T01:12:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2282617856 bytes 2024-02-10T01:12:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2330397696 bytes 2024-02-10T01:12:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2341298176 bytes 2024-02-10T01:12:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2362321920 bytes 2024-02-10T01:12:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2385097728 bytes 2024-02-10T01:12:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2464322560 bytes 2024-02-10T01:12:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2474061824 bytes 2024-02-10T01:12:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2475552768 bytes 2024-02-10T01:12:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2477788160 bytes 2024-02-10T01:12:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2482965504 bytes 2024-02-10T01:12:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2493237248 bytes 2024-02-10T01:12:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2549888000 bytes 2024-02-10T01:12:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2567969792 bytes 2024-02-10T01:12:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2592674816 bytes 2024-02-10T01:12:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2626987008 bytes 2024-02-10T01:12:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2682874880 bytes 2024-02-10T01:12:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2708705280 bytes 2024-02-10T01:12:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2749872128 bytes 2024-02-10T01:12:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2789912576 bytes 2024-02-10T01:12:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2812168192 bytes 2024-02-10T01:12:19.492 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-02-10T01:12:19.493 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:12:19.493 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-02-10T01:12:19.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-02-10T01:12:19.493 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-02-10T01:12:19.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-02-10T01:12:19.496 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-02-10T01:12:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2902279168 bytes 2024-02-10T01:12:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2906532864 bytes 2024-02-10T01:12:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2957882368 bytes 2024-02-10T01:12:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3012817920 bytes 2024-02-10T01:12:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3039203328 bytes 2024-02-10T01:12:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3061026816 bytes 2024-02-10T01:12:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3079665664 bytes 2024-02-10T01:12:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3081326592 bytes 2024-02-10T01:12:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3118480384 bytes 2024-02-10T01:12:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3139575808 bytes 2024-02-10T01:12:20.643 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:12:20.641+0000 7febc0775640 -1 reset not still connected to 0x561eda1091e0 2024-02-10T01:12:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3186945024 bytes 2024-02-10T01:12:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3201311744 bytes 2024-02-10T01:12:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3299462144 bytes 2024-02-10T01:12:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3344979968 bytes 2024-02-10T01:12:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3394950144 bytes 2024-02-10T01:12:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3446124544 bytes 2024-02-10T01:12:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3512024064 bytes 2024-02-10T01:12:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3554636800 bytes 2024-02-10T01:12:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3583212544 bytes 2024-02-10T01:12:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3642860544 bytes 2024-02-10T01:12:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3665526784 bytes 2024-02-10T01:12:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3705717760 bytes 2024-02-10T01:12:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:12:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:12:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:12:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:12:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:12:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:12:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:12:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:12:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:12:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:04 2024-02-10T01:12:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:12:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:12:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:12:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:12:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:12:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:12:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:04 2024-02-10T01:12:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:12:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-02-10T01:12:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:12:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:12:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:12:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:12:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:12:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:12:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:04 2024-02-10T01:12:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:12:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 8 2024-02-10T01:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:12:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:12:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:12:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:12:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:12:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:34 2024-02-10T01:12:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:12:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 8 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:12:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:12:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:12:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:12:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:12:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: down+unknown 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: status not found 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:12:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:34 2024-02-10T01:12:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-02-10T01:12:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 8 2024-02-10T01:12:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-02-10T01:12:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:12:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2024-02-10T01:12:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:test: 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527522,"remote_snapshot_timestamp":1707527522,"replay_state":"idle"} 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.3 on smithi060 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:45 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-02-10T01:12:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-02-10 01:12:34 2024-02-10T01:12:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:12:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527522,"remote_snapshot_timestamp":1707527522,"replay_state":"idle"} 2024-02-10T01:12:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:12:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:12:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-02-10T01:12:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:12:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:12:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.mirror_status 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'last_update=2024-02-10 01:12:45' 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-02-10 01:12:45' 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-02-10 01:12:45' 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror.mirror_status 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror.mirror_status 2024-02-10T01:12:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror.mirror_status 2024-02-10T01:12:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-02-10T01:12:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:12:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-02-10T01:12:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-02-10T01:12:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-02-10 01:12:45' 2024-02-10T01:12:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-02-10T01:12:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-02-10T01:12:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-02-10T01:12:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-02-10T01:12:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap10 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap10 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10 2024-02-10T01:12:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:12:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:12:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:12:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:12:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:12:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:12:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:12:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:12:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:12:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:12:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:12:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:12:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:12:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:12:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:12:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:12:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:12:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:12:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:12:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:12:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:12:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:12:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:12:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:12:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:12:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:12:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:12:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:12:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:12:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:12:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:12:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:12:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:12:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:12:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:12:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:12:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:12:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:12:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:12:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:12:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:12:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:12:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:12:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:12:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:12:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:12:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:12:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:12:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:12:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:12:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:12:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:12:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:12:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:12:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:12:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:12:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:12:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:12:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:12:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:12:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:12:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:12:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:12:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:12:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 33 2024-02-10T01:12:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:12:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:12:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:12:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:12:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:12:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=33 2024-02-10T01:12:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:12:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:12:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:12:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:12:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:12:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 33 = '' 2024-02-10T01:12:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:12:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:12:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:12:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:12:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:12:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 33 = '' 2024-02-10T01:12:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:12:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:12:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:12:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:12:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:12:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:12:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:12:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:12:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:12:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:12:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:12:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:12:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:12:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 33 = '' 2024-02-10T01:12:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:12:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:13:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:13:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:13:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:13:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=33 2024-02-10T01:13:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 33 = 33 2024-02-10T01:13:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:13:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:13:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-02-10T01:13:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:13:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:13:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:13:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 20 2024-02-10T01:13:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:12:08 2024 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:13:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:13:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:13:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:13:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:13:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:13:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snap9-clean ']' 2024-02-10T01:13:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap9-clean 2024-02-10T01:13:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9-clean 2024-02-10T01:13:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap10 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ clean_snap_name=snap10-clean 2024-02-10T01:13:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-02-10T01:13:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10-clean 2024-02-10T01:13:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-02-10T01:13:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-02-10T01:13:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:13:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:13:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror test started 2024-02-10T01:13:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:13:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:13:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:13:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:13:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:13:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:13:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:13:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-02-10T01:13:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:13:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:13:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:13:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:13:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:13:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:13:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:13:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:13:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:13:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-02-10T01:13:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:13:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:13:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:13:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:13:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-02-10T01:13:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:13:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:13:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:13:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:13:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:13:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:13:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:13:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:13:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:13:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-02-10T01:13:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-02-10T01:13:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:13:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:13:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:13:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:13:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:13:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:13:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:13:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:13:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:13:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:13:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:13:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:13:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-02-10T01:13:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:13:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:13:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:13:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:13:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:13:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-test.status 2024-02-10T01:13:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-test.status 2024-02-10T01:13:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:13:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:13:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-test.status 2024-02-10T01:13:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-test.status 2024-02-10T01:13:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:13:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-02-10T01:13:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-02-10T01:13:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 35 2024-02-10T01:13:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:13:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.status 2024-02-10T01:13:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-02-10T01:13:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:13:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:13:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=35 2024-02-10T01:13:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:13:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:13:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:13:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:13:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:13:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:13:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:13:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:13:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:13:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:13:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 35 = '' 2024-02-10T01:13:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:13:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.status 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-02-10T01:13:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:13:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=35 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 35 = 35 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10-clean 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:13:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:13:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-02-10T01:13:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'test': 2024-02-10T01:13:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 512 MiB in 128 objects 2024-02-10T01:13:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:13:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 21 2024-02-10T01:13:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10a6aa62bf4 2024-02-10T01:13:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10a6aa62bf4 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:12:08 2024 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:04:06 2024 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9f866f68-2640-41aa-9548-8fe0d3b85b7c 2024-02-10T01:13:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:13:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 10 2024-02-10T01:13:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap1 2024-02-10T01:13:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap1 2024-02-10T01:13:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1 2024-02-10T01:13:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap1-clean 2024-02-10T01:13:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap1-clean 2024-02-10T01:13:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap1-clean 2024-02-10T01:13:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap2 2024-02-10T01:13:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap2 2024-02-10T01:13:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap2-clean 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap2-clean 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap2-clean 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap3 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap3 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap3-clean 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap3-clean 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap3-clean 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap4 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap4 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap4-clean 2024-02-10T01:13:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap4-clean 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap4-clean 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap5 2024-02-10T01:13:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap5 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap5-clean 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap5-clean 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap5-clean 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap6 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap6 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap6-clean 2024-02-10T01:13:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap6-clean 2024-02-10T01:13:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap6-clean 2024-02-10T01:13:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap7 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap7 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap7-clean 2024-02-10T01:13:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap7-clean 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap7-clean 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap8 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap8 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap8-clean 2024-02-10T01:13:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap8-clean 2024-02-10T01:13:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap8-clean 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap9 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap9 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap9-clean 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap9-clean 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap9-clean 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap10 2024-02-10T01:13:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap10 2024-02-10T01:13:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:42.489 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-02-10T01:13:42.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-02-10T01:13:42.490 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-02-10T01:13:42.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:2024-02-10T01:13:42.490+0000 7f832713b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 39055) UID: 1000 2024-02-10T01:13:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap10-clean 2024-02-10T01:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror test snap10-clean 2024-02-10T01:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap10-clean 2024-02-10T01:13:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:13:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export 2024-02-10T01:13:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-02-10T01:13:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-test.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-test.export 2024-02-10T01:13:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:13:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 10 2024-02-10T01:13:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap1 2024-02-10T01:13:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-02-10T01:13:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap1 2024-02-10T01:13:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-02-10T01:13:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap1-clean 2024-02-10T01:13:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-02-10T01:13:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap1-clean 2024-02-10T01:13:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-02-10T01:13:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap2 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap2 2024-02-10T01:13:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-02-10T01:13:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap2-clean 2024-02-10T01:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-02-10T01:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap2-clean 2024-02-10T01:13:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-02-10T01:13:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap3 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap3 2024-02-10T01:13:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-02-10T01:13:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap3-clean 2024-02-10T01:13:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-02-10T01:13:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap3-clean 2024-02-10T01:13:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-02-10T01:13:49.491 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-02-10T01:13:49.491 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:13:49.492 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-02-10T01:13:49.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-02-10T01:13:49.492 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-02-10T01:13:49.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-02-10T01:13:49.495 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-02-10T01:13:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap4 2024-02-10T01:13:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-02-10T01:13:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap4 2024-02-10T01:13:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-02-10T01:13:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap4-clean 2024-02-10T01:13:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-02-10T01:13:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap4-clean 2024-02-10T01:13:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-02-10T01:13:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap5 2024-02-10T01:13:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-02-10T01:13:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap5 2024-02-10T01:13:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-02-10T01:13:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap5-clean 2024-02-10T01:13:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-02-10T01:13:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap5-clean 2024-02-10T01:13:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-02-10T01:13:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap6 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap6 2024-02-10T01:13:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-02-10T01:13:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap6-clean 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap6-clean 2024-02-10T01:13:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-02-10T01:13:55.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.283623015121511) > weight (0.25) 2024-02-10T01:13:55.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5102721934573015) > weight (0.25) 2024-02-10T01:13:55.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.265142701494987) > weight (0.25) 2024-02-10T01:13:55.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9265057751774523) > weight (0.25) 2024-02-10T01:13:55.497 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-02-10T01:13:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap7 2024-02-10T01:13:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-02-10T01:13:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap7 2024-02-10T01:13:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-02-10T01:13:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap7-clean 2024-02-10T01:13:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-02-10T01:13:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap7-clean 2024-02-10T01:13:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-02-10T01:13:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:13:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap8 2024-02-10T01:13:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-02-10T01:13:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap8 2024-02-10T01:13:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-02-10T01:13:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap8-clean 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap8-clean 2024-02-10T01:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-02-10T01:14:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:14:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:14:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap9 2024-02-10T01:14:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-02-10T01:14:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap9 2024-02-10T01:14:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-02-10T01:14:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:14:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap9-clean 2024-02-10T01:14:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-02-10T01:14:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap9-clean 2024-02-10T01:14:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-02-10T01:14:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:14:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 10` 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap10 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap10 2024-02-10T01:14:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-02-10T01:14:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap10-clean 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap=snap10-clean 2024-02-10T01:14:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-02-10T01:14:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing snap: 100% complete...done. 2024-02-10T01:14:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image_retry cluster2 mirror test 2024-02-10T01:14:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0 1 2 4 8 16 32 2024-02-10T01:14:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0 2024-02-10T01:14:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror test 2024-02-10T01:14:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-02-10T01:14:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-02-10T01:14:04.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi060.stderr:2024-02-10T01:14:04.336+0000 7fb777fc6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5594b7716680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-02-10T01:14:04.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi060.stderr:2024-02-10T01:14:04.336+0000 7fb77efd4640 -1 rbd::mirror::ImageReplayer: 0x5594b6977680 [3/9f866f68-2640-41aa-9548-8fe0d3b85b7c] operator(): start failed: (85) Interrupted system call should be restarted 2024-02-10T01:14:04.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi060.stderr:2024-02-10T01:14:04.463+0000 7fb777fc6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5594b73c2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-02-10T01:14:04.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi060.stderr:2024-02-10T01:14:04.463+0000 7fb77efd4640 -1 rbd::mirror::ImageReplayer: 0x5594b6977680 [3/9f866f68-2640-41aa-9548-8fe0d3b85b7c] operator(): start failed: (85) Interrupted system call should be restarted 2024-02-10T01:14:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-02-10T01:14:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-02-10T01:14:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:14:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=deleted 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image_id= 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n '' 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ get_image_id cluster1 mirror test 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local pool=mirror 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local image=test 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-02-10T01:14:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-02-10T01:14:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: error opening image test: (2) No such file or directory 2024-02-10T01:14:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image_id= 2024-02-10T01:14:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-02-10T01:14:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.1 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=test 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=deleted 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image_id= 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=deleted 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_image_id 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ get_image_id cluster1 mirror test 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:14:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local pool=mirror 2024-02-10T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local image=test 2024-02-10T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-02-10T01:14:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-02-10T01:14:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: error opening image test: (2) No such file or directory 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_image_id= 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n '' 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test deleted = deleted 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ testlog 'TEST: create many images' 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee -a /tmp/tmp.ywJfcqGoUP/rbd-mirror.test.log 2024-02-10T01:14:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ date '+%F %T' 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10 01:14:05 TEST: create many images 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10 01:14:05 TEST: create many images 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ snap_name=snap 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 50 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_1 2024-02-10T01:14:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_1 128M 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:14:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:14:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-02-10T01:14:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_1': 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1271bdfa9959 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1271bdfa9959 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:05 2024 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:05 2024 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:05 2024 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_1 2024-02-10T01:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=5 2024-02-10T01:14:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 37904384 bytes 2024-02-10T01:14:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 78141440 bytes 2024-02-10T01:14:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 101668864 bytes 2024-02-10T01:14:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 126683136 bytes 2024-02-10T01:14:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 165582848 bytes 2024-02-10T01:14:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 169645056 bytes 2024-02-10T01:14:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 204763136 bytes 2024-02-10T01:14:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 216482816 bytes 2024-02-10T01:14:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 335552512 bytes 2024-02-10T01:14:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 386362368 bytes 2024-02-10T01:14:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 474591232 bytes 2024-02-10T01:14:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 481764352 bytes 2024-02-10T01:14:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 528540672 bytes 2024-02-10T01:14:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 538120192 bytes 2024-02-10T01:14:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 636392448 bytes 2024-02-10T01:14:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 667608064 bytes 2024-02-10T01:14:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 734360576 bytes 2024-02-10T01:14:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 767807488 bytes 2024-02-10T01:14:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 840782848 bytes 2024-02-10T01:14:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 847210496 bytes 2024-02-10T01:14:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 863036416 bytes 2024-02-10T01:14:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 917833728 bytes 2024-02-10T01:14:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 943737856 bytes 2024-02-10T01:14:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 972721152 bytes 2024-02-10T01:14:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1019231232 bytes 2024-02-10T01:14:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1039910912 bytes 2024-02-10T01:14:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1096057856 bytes 2024-02-10T01:14:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1120710656 bytes 2024-02-10T01:14:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1188584448 bytes 2024-02-10T01:14:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1238819840 bytes 2024-02-10T01:14:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1258703872 bytes 2024-02-10T01:14:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1289399296 bytes 2024-02-10T01:14:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1335795712 bytes 2024-02-10T01:14:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1377594368 bytes 2024-02-10T01:14:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1394322432 bytes 2024-02-10T01:14:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1412547584 bytes 2024-02-10T01:14:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1498140672 bytes 2024-02-10T01:14:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1519478784 bytes 2024-02-10T01:14:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1566759936 bytes 2024-02-10T01:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1589494784 bytes 2024-02-10T01:14:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1605697536 bytes 2024-02-10T01:14:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1636031488 bytes 2024-02-10T01:14:10.923 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:14:10.922+0000 7fe71fd26640 -1 reset not still connected to 0x560a442a5d40 2024-02-10T01:14:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1685287936 bytes 2024-02-10T01:14:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1755966464 bytes 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_2 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_2 128M 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:14:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2024-02-10T01:14:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-02-10T01:14:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:14:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-02-10T01:14:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-02-10T01:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_2': 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 127dfa80bb93 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.127dfa80bb93 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:11 2024 2024-02-10T01:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:11 2024 2024-02-10T01:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:11 2024 2024-02-10T01:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:14:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_2 2024-02-10T01:14:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:14:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:14:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 41519104 bytes 2024-02-10T01:14:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 81095680 bytes 2024-02-10T01:14:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 109932544 bytes 2024-02-10T01:14:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 146990080 bytes 2024-02-10T01:14:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 196723712 bytes 2024-02-10T01:14:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 211631104 bytes 2024-02-10T01:14:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 237552640 bytes 2024-02-10T01:14:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 329072640 bytes 2024-02-10T01:14:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 349953024 bytes 2024-02-10T01:14:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 362225664 bytes 2024-02-10T01:14:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 404885504 bytes 2024-02-10T01:14:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 412221440 bytes 2024-02-10T01:14:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 468283392 bytes 2024-02-10T01:14:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 514588672 bytes 2024-02-10T01:14:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 595163136 bytes 2024-02-10T01:14:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 595799040 bytes 2024-02-10T01:14:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 646515712 bytes 2024-02-10T01:14:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 668454912 bytes 2024-02-10T01:14:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 697133056 bytes 2024-02-10T01:14:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 746461184 bytes 2024-02-10T01:14:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 813755392 bytes 2024-02-10T01:14:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 814365696 bytes 2024-02-10T01:14:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 865185792 bytes 2024-02-10T01:14:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 890695680 bytes 2024-02-10T01:14:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 911289344 bytes 2024-02-10T01:14:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1039888384 bytes 2024-02-10T01:14:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1062550528 bytes 2024-02-10T01:14:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1111233536 bytes 2024-02-10T01:14:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1161548800 bytes 2024-02-10T01:14:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1213990912 bytes 2024-02-10T01:14:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1262041088 bytes 2024-02-10T01:14:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1281095680 bytes 2024-02-10T01:14:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1336568832 bytes 2024-02-10T01:14:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1402148864 bytes 2024-02-10T01:14:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1428512768 bytes 2024-02-10T01:14:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1490177024 bytes 2024-02-10T01:14:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1565759488 bytes 2024-02-10T01:14:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1591677952 bytes 2024-02-10T01:14:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1682664448 bytes 2024-02-10T01:14:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1701618688 bytes 2024-02-10T01:14:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1778523136 bytes 2024-02-10T01:14:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1779134464 bytes 2024-02-10T01:14:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1821190144 bytes 2024-02-10T01:14:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1844950016 bytes 2024-02-10T01:14:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1853053952 bytes 2024-02-10T01:14:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1853639680 bytes 2024-02-10T01:14:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1893586944 bytes 2024-02-10T01:14:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1922916352 bytes 2024-02-10T01:14:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1979891712 bytes 2024-02-10T01:14:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2010757120 bytes 2024-02-10T01:14:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2057298944 bytes 2024-02-10T01:14:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2107905024 bytes 2024-02-10T01:14:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2166027264 bytes 2024-02-10T01:14:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2174382080 bytes 2024-02-10T01:14:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2229947392 bytes 2024-02-10T01:14:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2234769408 bytes 2024-02-10T01:14:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2285683712 bytes 2024-02-10T01:14:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2329241600 bytes 2024-02-10T01:14:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2368625664 bytes 2024-02-10T01:14:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2406311936 bytes 2024-02-10T01:14:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2446548992 bytes 2024-02-10T01:14:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2478140416 bytes 2024-02-10T01:14:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2512006144 bytes 2024-02-10T01:14:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2519845888 bytes 2024-02-10T01:14:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2551510016 bytes 2024-02-10T01:14:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2578337792 bytes 2024-02-10T01:14:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2578948096 bytes 2024-02-10T01:14:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2631351296 bytes 2024-02-10T01:14:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2656198656 bytes 2024-02-10T01:14:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2687882240 bytes 2024-02-10T01:14:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2728743936 bytes 2024-02-10T01:14:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2750989312 bytes 2024-02-10T01:14:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_3 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_3 128M 2024-02-10T01:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2024-02-10T01:14:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-02-10T01:14:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_3': 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1289a478c9dc 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1289a478c9dc 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:20 2024 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:20 2024 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:20 2024 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:14:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_3 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 32323584 bytes 2024-02-10T01:14:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 63279104 bytes 2024-02-10T01:14:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 124582912 bytes 2024-02-10T01:14:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 149176320 bytes 2024-02-10T01:14:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 149799936 bytes 2024-02-10T01:14:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 192322560 bytes 2024-02-10T01:14:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 239537152 bytes 2024-02-10T01:14:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 277691392 bytes 2024-02-10T01:14:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 349212672 bytes 2024-02-10T01:14:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 449822720 bytes 2024-02-10T01:14:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 476113920 bytes 2024-02-10T01:14:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 552278016 bytes 2024-02-10T01:14:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 582654976 bytes 2024-02-10T01:14:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 585335808 bytes 2024-02-10T01:14:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 595404800 bytes 2024-02-10T01:14:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 664123392 bytes 2024-02-10T01:14:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 719889408 bytes 2024-02-10T01:14:23.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 761695232 bytes 2024-02-10T01:14:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 817630208 bytes 2024-02-10T01:14:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 856017920 bytes 2024-02-10T01:14:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 913925120 bytes 2024-02-10T01:14:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 940324864 bytes 2024-02-10T01:14:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1050993664 bytes 2024-02-10T01:14:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1065323520 bytes 2024-02-10T01:14:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1123919872 bytes 2024-02-10T01:14:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1130459136 bytes 2024-02-10T01:14:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1143503872 bytes 2024-02-10T01:14:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1160675328 bytes 2024-02-10T01:14:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1197360128 bytes 2024-02-10T01:14:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1237735424 bytes 2024-02-10T01:14:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1271712768 bytes 2024-02-10T01:14:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1304129536 bytes 2024-02-10T01:14:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1346427904 bytes 2024-02-10T01:14:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1394052096 bytes 2024-02-10T01:14:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1440372736 bytes 2024-02-10T01:14:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1494758400 bytes 2024-02-10T01:14:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1536562176 bytes 2024-02-10T01:14:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1557376000 bytes 2024-02-10T01:14:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1564085248 bytes 2024-02-10T01:14:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1614477312 bytes 2024-02-10T01:14:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1615118336 bytes 2024-02-10T01:14:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1669254144 bytes 2024-02-10T01:14:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1764762624 bytes 2024-02-10T01:14:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1862758400 bytes 2024-02-10T01:14:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1906668544 bytes 2024-02-10T01:14:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1932123136 bytes 2024-02-10T01:14:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1989227520 bytes 2024-02-10T01:14:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2039870464 bytes 2024-02-10T01:14:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2102002688 bytes 2024-02-10T01:14:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2135849984 bytes 2024-02-10T01:14:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2214313984 bytes 2024-02-10T01:14:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2236019712 bytes 2024-02-10T01:14:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2267923456 bytes 2024-02-10T01:14:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2285207552 bytes 2024-02-10T01:14:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2341272576 bytes 2024-02-10T01:14:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2383966208 bytes 2024-02-10T01:14:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2419809280 bytes 2024-02-10T01:14:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2423214080 bytes 2024-02-10T01:14:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2487077888 bytes 2024-02-10T01:14:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_4 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_4 128M 2024-02-10T01:14:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:14:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-02-10T01:14:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-02-10T01:14:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_4': 2024-02-10T01:14:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295667ee071 2024-02-10T01:14:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295667ee071 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:28 2024 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:28 2024 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:28 2024 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:14:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_4 2024-02-10T01:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=6 2024-02-10T01:14:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 16237568 bytes 2024-02-10T01:14:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 37676032 bytes 2024-02-10T01:14:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 38291456 bytes 2024-02-10T01:14:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 58151936 bytes 2024-02-10T01:14:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 80535552 bytes 2024-02-10T01:14:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 146249728 bytes 2024-02-10T01:14:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 191713280 bytes 2024-02-10T01:14:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 245054464 bytes 2024-02-10T01:14:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 320612352 bytes 2024-02-10T01:14:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 364608512 bytes 2024-02-10T01:14:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 424826880 bytes 2024-02-10T01:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 483085312 bytes 2024-02-10T01:14:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 521668608 bytes 2024-02-10T01:14:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 535885824 bytes 2024-02-10T01:14:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 545138688 bytes 2024-02-10T01:14:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 545739776 bytes 2024-02-10T01:14:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 566534144 bytes 2024-02-10T01:14:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 659996672 bytes 2024-02-10T01:14:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 689939456 bytes 2024-02-10T01:14:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 764752896 bytes 2024-02-10T01:14:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 789212160 bytes 2024-02-10T01:14:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 789823488 bytes 2024-02-10T01:14:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 815849472 bytes 2024-02-10T01:14:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 865358848 bytes 2024-02-10T01:14:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 926584832 bytes 2024-02-10T01:14:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 947972096 bytes 2024-02-10T01:14:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 972512256 bytes 2024-02-10T01:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1032172544 bytes 2024-02-10T01:14:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1065890816 bytes 2024-02-10T01:14:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1099101184 bytes 2024-02-10T01:14:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1103383552 bytes 2024-02-10T01:14:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1146454016 bytes 2024-02-10T01:14:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1187211264 bytes 2024-02-10T01:14:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1218761728 bytes 2024-02-10T01:14:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1320537088 bytes 2024-02-10T01:14:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1382573056 bytes 2024-02-10T01:14:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1399798784 bytes 2024-02-10T01:14:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1415694336 bytes 2024-02-10T01:14:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1449289728 bytes 2024-02-10T01:14:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1512708096 bytes 2024-02-10T01:14:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1539015680 bytes 2024-02-10T01:14:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1579376640 bytes 2024-02-10T01:14:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1601445888 bytes 2024-02-10T01:14:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1628627968 bytes 2024-02-10T01:14:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1680766976 bytes 2024-02-10T01:14:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1724496896 bytes 2024-02-10T01:14:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1774005248 bytes 2024-02-10T01:14:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1832584192 bytes 2024-02-10T01:14:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1873123328 bytes 2024-02-10T01:14:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1915947008 bytes 2024-02-10T01:14:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1939192832 bytes 2024-02-10T01:14:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2023819264 bytes 2024-02-10T01:14:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2065710080 bytes 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_5 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_5 128M 2024-02-10T01:14:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-02-10T01:14:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_5': 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12a1842f904a 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12a1842f904a 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:35 2024 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:35 2024 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:35 2024 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:14:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_5 2024-02-10T01:14:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:14:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=5 2024-02-10T01:14:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 74749952 bytes 2024-02-10T01:14:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 96190464 bytes 2024-02-10T01:14:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 243851264 bytes 2024-02-10T01:14:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 313657344 bytes 2024-02-10T01:14:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 336901120 bytes 2024-02-10T01:14:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 418759680 bytes 2024-02-10T01:14:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 486538240 bytes 2024-02-10T01:14:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 595195904 bytes 2024-02-10T01:14:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 635524096 bytes 2024-02-10T01:14:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 673314816 bytes 2024-02-10T01:14:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 682490880 bytes 2024-02-10T01:14:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 728622080 bytes 2024-02-10T01:14:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 785630208 bytes 2024-02-10T01:14:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 809502720 bytes 2024-02-10T01:14:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 844925952 bytes 2024-02-10T01:14:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 883980288 bytes 2024-02-10T01:14:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 926449664 bytes 2024-02-10T01:14:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 1009391616 bytes 2024-02-10T01:14:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 1054846976 bytes 2024-02-10T01:14:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1099686912 bytes 2024-02-10T01:14:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1127508992 bytes 2024-02-10T01:14:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1171666944 bytes 2024-02-10T01:14:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1236382720 bytes 2024-02-10T01:14:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1293277184 bytes 2024-02-10T01:14:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1335535616 bytes 2024-02-10T01:14:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1394767872 bytes 2024-02-10T01:14:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1425385472 bytes 2024-02-10T01:14:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1485395968 bytes 2024-02-10T01:14:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1565281280 bytes 2024-02-10T01:14:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1588024320 bytes 2024-02-10T01:14:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1640091648 bytes 2024-02-10T01:14:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1651930112 bytes 2024-02-10T01:14:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1697721344 bytes 2024-02-10T01:14:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1774237696 bytes 2024-02-10T01:14:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1823647744 bytes 2024-02-10T01:14:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1875401728 bytes 2024-02-10T01:14:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1978451968 bytes 2024-02-10T01:14:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_6 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_6 128M 2024-02-10T01:14:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-02-10T01:14:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-02-10T01:14:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_6': 2024-02-10T01:14:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12add8471330 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12add8471330 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:41 2024 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:41 2024 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:41 2024 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_6 2024-02-10T01:14:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:14:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=6 2024-02-10T01:14:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 32388096 bytes 2024-02-10T01:14:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 52495360 bytes 2024-02-10T01:14:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 98310144 bytes 2024-02-10T01:14:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 118782976 bytes 2024-02-10T01:14:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 176382976 bytes 2024-02-10T01:14:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 204675072 bytes 2024-02-10T01:14:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 223401984 bytes 2024-02-10T01:14:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 274985984 bytes 2024-02-10T01:14:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 312030208 bytes 2024-02-10T01:14:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 312645632 bytes 2024-02-10T01:14:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 326501376 bytes 2024-02-10T01:14:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 379145216 bytes 2024-02-10T01:14:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 395768832 bytes 2024-02-10T01:14:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 420580352 bytes 2024-02-10T01:14:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 443372544 bytes 2024-02-10T01:14:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 513852416 bytes 2024-02-10T01:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 574132224 bytes 2024-02-10T01:14:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 597176320 bytes 2024-02-10T01:14:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 671148032 bytes 2024-02-10T01:14:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 731587584 bytes 2024-02-10T01:14:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 752972800 bytes 2024-02-10T01:14:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 829956096 bytes 2024-02-10T01:14:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 854466560 bytes 2024-02-10T01:14:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 886087680 bytes 2024-02-10T01:14:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 929684480 bytes 2024-02-10T01:14:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 992785408 bytes 2024-02-10T01:14:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1018645504 bytes 2024-02-10T01:14:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1054525440 bytes 2024-02-10T01:14:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1119646720 bytes 2024-02-10T01:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1126061056 bytes 2024-02-10T01:14:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1168090112 bytes 2024-02-10T01:14:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1182168064 bytes 2024-02-10T01:14:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1210888192 bytes 2024-02-10T01:14:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1214528512 bytes 2024-02-10T01:14:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1239026688 bytes 2024-02-10T01:14:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1272762368 bytes 2024-02-10T01:14:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1339143168 bytes 2024-02-10T01:14:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1371158528 bytes 2024-02-10T01:14:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1409616896 bytes 2024-02-10T01:14:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1464685568 bytes 2024-02-10T01:14:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1522423808 bytes 2024-02-10T01:14:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1564818432 bytes 2024-02-10T01:14:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1588506624 bytes 2024-02-10T01:14:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1615097856 bytes 2024-02-10T01:14:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1684869120 bytes 2024-02-10T01:14:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1753810944 bytes 2024-02-10T01:14:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1767158784 bytes 2024-02-10T01:14:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1843737600 bytes 2024-02-10T01:14:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1878276096 bytes 2024-02-10T01:14:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1924860928 bytes 2024-02-10T01:14:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1954608128 bytes 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_7 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_7 128M 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2024-02-10T01:14:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-02-10T01:14:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-02-10T01:14:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_7': 2024-02-10T01:14:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12b98c05febb 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12b98c05febb 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:47 2024 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:47 2024 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:47 2024 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:14:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_7 2024-02-10T01:14:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:14:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=6 2024-02-10T01:14:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 48653312 bytes 2024-02-10T01:14:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 61710336 bytes 2024-02-10T01:14:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 147832832 bytes 2024-02-10T01:14:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 169702400 bytes 2024-02-10T01:14:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 222071808 bytes 2024-02-10T01:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 237408256 bytes 2024-02-10T01:14:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 256208896 bytes 2024-02-10T01:14:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 284160000 bytes 2024-02-10T01:14:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 331667456 bytes 2024-02-10T01:14:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 361306112 bytes 2024-02-10T01:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 414893056 bytes 2024-02-10T01:14:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 473675776 bytes 2024-02-10T01:14:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 512386048 bytes 2024-02-10T01:14:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 547492864 bytes 2024-02-10T01:14:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 595534848 bytes 2024-02-10T01:14:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 623922176 bytes 2024-02-10T01:14:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 678990848 bytes 2024-02-10T01:14:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 758717440 bytes 2024-02-10T01:14:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 816952320 bytes 2024-02-10T01:14:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 849032192 bytes 2024-02-10T01:14:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 893125632 bytes 2024-02-10T01:14:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 998177792 bytes 2024-02-10T01:14:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1060750336 bytes 2024-02-10T01:14:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1165903872 bytes 2024-02-10T01:14:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1219894272 bytes 2024-02-10T01:14:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1296239616 bytes 2024-02-10T01:14:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1334445056 bytes 2024-02-10T01:14:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1424991232 bytes 2024-02-10T01:14:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1467499520 bytes 2024-02-10T01:14:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1488084992 bytes 2024-02-10T01:14:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1501898752 bytes 2024-02-10T01:14:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1542803456 bytes 2024-02-10T01:14:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1543426048 bytes 2024-02-10T01:14:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1633180672 bytes 2024-02-10T01:14:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1753203712 bytes 2024-02-10T01:14:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1826837504 bytes 2024-02-10T01:14:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1874528256 bytes 2024-02-10T01:14:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1951949824 bytes 2024-02-10T01:14:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1996581888 bytes 2024-02-10T01:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 2026730496 bytes 2024-02-10T01:14:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 2038866944 bytes 2024-02-10T01:14:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 2121478144 bytes 2024-02-10T01:14:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2177600512 bytes 2024-02-10T01:14:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2203031552 bytes 2024-02-10T01:14:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2263251968 bytes 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_8 2024-02-10T01:14:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_8 128M 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:14:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2024-02-10T01:14:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:14:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-02-10T01:14:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_8': 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12c5191a222f 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12c5191a222f 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:14:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:54 2024 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:54 2024 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:54 2024 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:14:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:14:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:14:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_8 2024-02-10T01:14:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:14:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:14:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:14:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:14:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=10 2024-02-10T01:14:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:14:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:14:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 20997120 bytes 2024-02-10T01:14:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 37149696 bytes 2024-02-10T01:14:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 111156224 bytes 2024-02-10T01:14:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 161466368 bytes 2024-02-10T01:14:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 198626304 bytes 2024-02-10T01:14:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 199243776 bytes 2024-02-10T01:14:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 229317632 bytes 2024-02-10T01:14:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 276471808 bytes 2024-02-10T01:14:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 307645440 bytes 2024-02-10T01:14:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 324408320 bytes 2024-02-10T01:14:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 368156672 bytes 2024-02-10T01:14:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 421585920 bytes 2024-02-10T01:14:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 463789056 bytes 2024-02-10T01:14:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 509389824 bytes 2024-02-10T01:14:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 550908928 bytes 2024-02-10T01:14:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 619486208 bytes 2024-02-10T01:14:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 624339968 bytes 2024-02-10T01:14:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 674690048 bytes 2024-02-10T01:14:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 688362496 bytes 2024-02-10T01:14:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 718235648 bytes 2024-02-10T01:14:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 749556736 bytes 2024-02-10T01:14:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 753318912 bytes 2024-02-10T01:14:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 842995712 bytes 2024-02-10T01:14:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 881289216 bytes 2024-02-10T01:14:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 958971904 bytes 2024-02-10T01:14:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1001999360 bytes 2024-02-10T01:14:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1063321600 bytes 2024-02-10T01:14:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1080513536 bytes 2024-02-10T01:14:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1102673920 bytes 2024-02-10T01:14:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1233107968 bytes 2024-02-10T01:14:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1254828032 bytes 2024-02-10T01:14:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1304176640 bytes 2024-02-10T01:14:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1339384832 bytes 2024-02-10T01:14:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1362874368 bytes 2024-02-10T01:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1418157056 bytes 2024-02-10T01:14:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1462280192 bytes 2024-02-10T01:14:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1550950400 bytes 2024-02-10T01:14:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1575014400 bytes 2024-02-10T01:15:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1659715584 bytes 2024-02-10T01:15:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1717824512 bytes 2024-02-10T01:15:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1762572288 bytes 2024-02-10T01:15:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1810114560 bytes 2024-02-10T01:15:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1865801728 bytes 2024-02-10T01:15:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1903261696 bytes 2024-02-10T01:15:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1931537408 bytes 2024-02-10T01:15:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1969561600 bytes 2024-02-10T01:15:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1999579136 bytes 2024-02-10T01:15:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2029997056 bytes 2024-02-10T01:15:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2055099392 bytes 2024-02-10T01:15:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2127915008 bytes 2024-02-10T01:15:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2215279616 bytes 2024-02-10T01:15:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2301436928 bytes 2024-02-10T01:15:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2360137728 bytes 2024-02-10T01:15:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2421789696 bytes 2024-02-10T01:15:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2508251136 bytes 2024-02-10T01:15:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2535351296 bytes 2024-02-10T01:15:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2583635968 bytes 2024-02-10T01:15:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2649138176 bytes 2024-02-10T01:15:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2698201088 bytes 2024-02-10T01:15:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2748258304 bytes 2024-02-10T01:15:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2805488640 bytes 2024-02-10T01:15:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2825675776 bytes 2024-02-10T01:15:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2836158464 bytes 2024-02-10T01:15:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2900465664 bytes 2024-02-10T01:15:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2925801472 bytes 2024-02-10T01:15:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3016971264 bytes 2024-02-10T01:15:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3062941696 bytes 2024-02-10T01:15:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3116127232 bytes 2024-02-10T01:15:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3149356032 bytes 2024-02-10T01:15:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3189513216 bytes 2024-02-10T01:15:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3308980224 bytes 2024-02-10T01:15:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3331628032 bytes 2024-02-10T01:15:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3378414592 bytes 2024-02-10T01:15:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3481558016 bytes 2024-02-10T01:15:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3531125760 bytes 2024-02-10T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:15:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_9 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_9 128M 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:15:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-02-10T01:15:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:15:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-02-10T01:15:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_9': 2024-02-10T01:15:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:15:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:15:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:15:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12d1bf4775fc 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12d1bf4775fc 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:05 2024 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:05 2024 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:05 2024 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:15:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:15:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:15:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_9 2024-02-10T01:15:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:15:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:15:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:15:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:15:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:15:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 45801472 bytes 2024-02-10T01:15:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 151229440 bytes 2024-02-10T01:15:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 175995904 bytes 2024-02-10T01:15:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 215596032 bytes 2024-02-10T01:15:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 307836928 bytes 2024-02-10T01:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 384413696 bytes 2024-02-10T01:15:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 412676096 bytes 2024-02-10T01:15:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 511947776 bytes 2024-02-10T01:15:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 521009152 bytes 2024-02-10T01:15:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 540634112 bytes 2024-02-10T01:15:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 574047232 bytes 2024-02-10T01:15:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 601339904 bytes 2024-02-10T01:15:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 620612608 bytes 2024-02-10T01:15:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 670583808 bytes 2024-02-10T01:15:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 689818624 bytes 2024-02-10T01:15:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 726586368 bytes 2024-02-10T01:15:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 739534848 bytes 2024-02-10T01:15:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 764947456 bytes 2024-02-10T01:15:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 819646464 bytes 2024-02-10T01:15:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 851643392 bytes 2024-02-10T01:15:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 879132672 bytes 2024-02-10T01:15:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 919740416 bytes 2024-02-10T01:15:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 965140480 bytes 2024-02-10T01:15:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 980281344 bytes 2024-02-10T01:15:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1033819136 bytes 2024-02-10T01:15:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1077266432 bytes 2024-02-10T01:15:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1094604800 bytes 2024-02-10T01:15:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1124989952 bytes 2024-02-10T01:15:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1201014784 bytes 2024-02-10T01:15:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1224818688 bytes 2024-02-10T01:15:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1270479872 bytes 2024-02-10T01:15:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1296157696 bytes 2024-02-10T01:15:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1342479360 bytes 2024-02-10T01:15:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1355128832 bytes 2024-02-10T01:15:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1397678080 bytes 2024-02-10T01:15:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1415386112 bytes 2024-02-10T01:15:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1447707648 bytes 2024-02-10T01:15:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1497119744 bytes 2024-02-10T01:15:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1511432192 bytes 2024-02-10T01:15:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1573233664 bytes 2024-02-10T01:15:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1577708544 bytes 2024-02-10T01:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1600452608 bytes 2024-02-10T01:15:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1643871232 bytes 2024-02-10T01:15:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1757289472 bytes 2024-02-10T01:15:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1792385024 bytes 2024-02-10T01:15:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1830519808 bytes 2024-02-10T01:15:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1870220288 bytes 2024-02-10T01:15:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1930566656 bytes 2024-02-10T01:15:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1960441856 bytes 2024-02-10T01:15:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1990956032 bytes 2024-02-10T01:15:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2051691520 bytes 2024-02-10T01:15:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2087433216 bytes 2024-02-10T01:15:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2132454400 bytes 2024-02-10T01:15:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2206822400 bytes 2024-02-10T01:15:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2215892992 bytes 2024-02-10T01:15:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2253366272 bytes 2024-02-10T01:15:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2283169792 bytes 2024-02-10T01:15:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2316082176 bytes 2024-02-10T01:15:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2347187200 bytes 2024-02-10T01:15:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2380512256 bytes 2024-02-10T01:15:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2405390336 bytes 2024-02-10T01:15:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2479054848 bytes 2024-02-10T01:15:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2549952512 bytes 2024-02-10T01:15:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2602613760 bytes 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_10 2024-02-10T01:15:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_10 128M 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-02-10T01:15:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:15:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_10': 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12dda458d954 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12dda458d954 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:15:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:14 2024 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:14 2024 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:14 2024 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:15:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:15:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:15:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_10 2024-02-10T01:15:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:15:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:15:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:15:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:15:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:15:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 55736320 bytes 2024-02-10T01:15:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 83678208 bytes 2024-02-10T01:15:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 135757824 bytes 2024-02-10T01:15:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 184928256 bytes 2024-02-10T01:15:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 239822848 bytes 2024-02-10T01:15:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 305087488 bytes 2024-02-10T01:15:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 354356224 bytes 2024-02-10T01:15:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 427400192 bytes 2024-02-10T01:15:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 437160960 bytes 2024-02-10T01:15:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 499930112 bytes 2024-02-10T01:15:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 542964736 bytes 2024-02-10T01:15:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 606577664 bytes 2024-02-10T01:15:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 649954304 bytes 2024-02-10T01:15:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 663754752 bytes 2024-02-10T01:15:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 757396480 bytes 2024-02-10T01:15:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 770311168 bytes 2024-02-10T01:15:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 811492352 bytes 2024-02-10T01:15:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 821909504 bytes 2024-02-10T01:15:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 833531904 bytes 2024-02-10T01:15:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 842806272 bytes 2024-02-10T01:15:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 929124352 bytes 2024-02-10T01:15:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 962096128 bytes 2024-02-10T01:15:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1053860864 bytes 2024-02-10T01:15:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1075597312 bytes 2024-02-10T01:15:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1120962560 bytes 2024-02-10T01:15:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1186532352 bytes 2024-02-10T01:15:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1297150976 bytes 2024-02-10T01:15:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1339496448 bytes 2024-02-10T01:15:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1362484224 bytes 2024-02-10T01:15:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1415930880 bytes 2024-02-10T01:15:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1438713856 bytes 2024-02-10T01:15:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1503037440 bytes 2024-02-10T01:15:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1538581504 bytes 2024-02-10T01:15:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1583676416 bytes 2024-02-10T01:15:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1587390464 bytes 2024-02-10T01:15:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1643156480 bytes 2024-02-10T01:15:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1653833728 bytes 2024-02-10T01:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1674531840 bytes 2024-02-10T01:15:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1713572864 bytes 2024-02-10T01:15:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1792095232 bytes 2024-02-10T01:15:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1806664704 bytes 2024-02-10T01:15:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1882791936 bytes 2024-02-10T01:15:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1912614912 bytes 2024-02-10T01:15:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2035250176 bytes 2024-02-10T01:15:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2083215360 bytes 2024-02-10T01:15:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2143616000 bytes 2024-02-10T01:15:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2177009664 bytes 2024-02-10T01:15:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2191270912 bytes 2024-02-10T01:15:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2234190848 bytes 2024-02-10T01:15:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2245026816 bytes 2024-02-10T01:15:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2266043392 bytes 2024-02-10T01:15:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2283629568 bytes 2024-02-10T01:15:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2326396928 bytes 2024-02-10T01:15:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2396865536 bytes 2024-02-10T01:15:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2465153024 bytes 2024-02-10T01:15:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2531096576 bytes 2024-02-10T01:15:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2564365312 bytes 2024-02-10T01:15:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2579940352 bytes 2024-02-10T01:15:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2640261120 bytes 2024-02-10T01:15:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2663213056 bytes 2024-02-10T01:15:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2732149760 bytes 2024-02-10T01:15:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2871114752 bytes 2024-02-10T01:15:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:15:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:15:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:15:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_11 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_11 128M 2024-02-10T01:15:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-02-10T01:15:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:15:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-02-10T01:15:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_11': 2024-02-10T01:15:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12e9c57409f7 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12e9c57409f7 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:23 2024 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:23 2024 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:23 2024 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_11 2024-02-10T01:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:15:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 24219648 bytes 2024-02-10T01:15:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 94637056 bytes 2024-02-10T01:15:24.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 174128128 bytes 2024-02-10T01:15:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 278459392 bytes 2024-02-10T01:15:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 347549696 bytes 2024-02-10T01:15:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 361392128 bytes 2024-02-10T01:15:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 432631808 bytes 2024-02-10T01:15:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 460476416 bytes 2024-02-10T01:15:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 488624128 bytes 2024-02-10T01:15:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 523234304 bytes 2024-02-10T01:15:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 556800000 bytes 2024-02-10T01:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 584687616 bytes 2024-02-10T01:15:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 597858304 bytes 2024-02-10T01:15:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 604531712 bytes 2024-02-10T01:15:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 646945792 bytes 2024-02-10T01:15:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 685006848 bytes 2024-02-10T01:15:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 736702464 bytes 2024-02-10T01:15:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 775641088 bytes 2024-02-10T01:15:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 826328064 bytes 2024-02-10T01:15:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 858706944 bytes 2024-02-10T01:15:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 883938304 bytes 2024-02-10T01:15:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 901943296 bytes 2024-02-10T01:15:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 924166144 bytes 2024-02-10T01:15:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 981336064 bytes 2024-02-10T01:15:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1000779776 bytes 2024-02-10T01:15:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1035967488 bytes 2024-02-10T01:15:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1086579712 bytes 2024-02-10T01:15:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1127190528 bytes 2024-02-10T01:15:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1169559552 bytes 2024-02-10T01:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1275641856 bytes 2024-02-10T01:15:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1306224640 bytes 2024-02-10T01:15:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1344769024 bytes 2024-02-10T01:15:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1394653184 bytes 2024-02-10T01:15:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1472529408 bytes 2024-02-10T01:15:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1509638144 bytes 2024-02-10T01:15:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1573960704 bytes 2024-02-10T01:15:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1574583296 bytes 2024-02-10T01:15:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1615812608 bytes 2024-02-10T01:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1644175360 bytes 2024-02-10T01:15:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1654103040 bytes 2024-02-10T01:15:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1686149120 bytes 2024-02-10T01:15:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1691536384 bytes 2024-02-10T01:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1743689728 bytes 2024-02-10T01:15:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1802573824 bytes 2024-02-10T01:15:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1873861632 bytes 2024-02-10T01:15:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1925295104 bytes 2024-02-10T01:15:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1964509184 bytes 2024-02-10T01:15:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1968205824 bytes 2024-02-10T01:15:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2084581376 bytes 2024-02-10T01:15:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2114711552 bytes 2024-02-10T01:15:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2123160576 bytes 2024-02-10T01:15:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2132794368 bytes 2024-02-10T01:15:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2165269504 bytes 2024-02-10T01:15:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2185361408 bytes 2024-02-10T01:15:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2239222784 bytes 2024-02-10T01:15:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2271685632 bytes 2024-02-10T01:15:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2306440192 bytes 2024-02-10T01:15:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2359260160 bytes 2024-02-10T01:15:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2418292736 bytes 2024-02-10T01:15:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2445998080 bytes 2024-02-10T01:15:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2480436224 bytes 2024-02-10T01:15:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2502704128 bytes 2024-02-10T01:15:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2527614976 bytes 2024-02-10T01:15:31.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2562748416 bytes 2024-02-10T01:15:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2589361152 bytes 2024-02-10T01:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2624825344 bytes 2024-02-10T01:15:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2646167552 bytes 2024-02-10T01:15:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2674325504 bytes 2024-02-10T01:15:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2699810816 bytes 2024-02-10T01:15:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2747039744 bytes 2024-02-10T01:15:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2747667456 bytes 2024-02-10T01:15:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2780943360 bytes 2024-02-10T01:15:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2809587712 bytes 2024-02-10T01:15:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 2912475136 bytes 2024-02-10T01:15:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 2932110336 bytes 2024-02-10T01:15:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 2948491264 bytes 2024-02-10T01:15:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 2986789888 bytes 2024-02-10T01:15:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3019157504 bytes 2024-02-10T01:15:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3060167680 bytes 2024-02-10T01:15:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3128282112 bytes 2024-02-10T01:15:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3128868864 bytes 2024-02-10T01:15:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3162515456 bytes 2024-02-10T01:15:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3245185024 bytes 2024-02-10T01:15:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3269388288 bytes 2024-02-10T01:15:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3325243392 bytes 2024-02-10T01:15:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3361140736 bytes 2024-02-10T01:15:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3388423168 bytes 2024-02-10T01:15:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3425782784 bytes 2024-02-10T01:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3449869312 bytes 2024-02-10T01:15:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3466967040 bytes 2024-02-10T01:15:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3497571328 bytes 2024-02-10T01:15:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3555832832 bytes 2024-02-10T01:15:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3561779200 bytes 2024-02-10T01:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3599503360 bytes 2024-02-10T01:15:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3625169920 bytes 2024-02-10T01:15:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3643277312 bytes 2024-02-10T01:15:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 3677224960 bytes 2024-02-10T01:15:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 3732975616 bytes 2024-02-10T01:15:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 3788496896 bytes 2024-02-10T01:15:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 3820139520 bytes 2024-02-10T01:15:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 3848462336 bytes 2024-02-10T01:15:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 3922557952 bytes 2024-02-10T01:15:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 3945860096 bytes 2024-02-10T01:15:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 3995673600 bytes 2024-02-10T01:15:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10500 IOs, 4037079040 bytes 2024-02-10T01:15:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10600 IOs, 4123664384 bytes 2024-02-10T01:15:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10700 IOs, 4128155648 bytes 2024-02-10T01:15:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10800 IOs, 4134260736 bytes 2024-02-10T01:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_12 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_12 128M 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:15:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:15:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:15:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-02-10T01:15:37.499 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6364645640806446) > weight (0.25) 2024-02-10T01:15:37.499 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2827850107835156) > weight (0.25) 2024-02-10T01:15:37.499 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.479925951572153) > weight (0.25) 2024-02-10T01:15:37.499 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-02-10T01:15:37.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-02-10T01:15:37.499 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-02-10T01:15:37.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi060.stderr:2024-02-10T01:15:37.499+0000 7f7e26f8e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 38700) UID: 1000 2024-02-10T01:15:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:15:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-02-10T01:15:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_12': 2024-02-10T01:15:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12f5a4e74733 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12f5a4e74733 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:37 2024 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:37 2024 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:37 2024 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:15:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_12 2024-02-10T01:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:15:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:15:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:15:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:15:38.501 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-02-10T01:15:38.501 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:15:38.501 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-02-10T01:15:38.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-02-10T01:15:38.502 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-02-10T01:15:38.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-02-10T01:15:38.505 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-02-10T01:15:38.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:15:38.595+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823c280 [3/4f363a97-9a20-4913-a174-4d1fab8a425e] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:15:38.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:15:38.595+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823c280 [3/4f363a97-9a20-4913-a174-4d1fab8a425e] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:15:38.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:15:38.617+0000 7ff49842c640 -1 rbd::mirror::ImageReplayer: 0x55c392258280 [3/69e299aa-5815-4202-b2c2-9fd34a982ad9] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:15:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 86138880 bytes 2024-02-10T01:15:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 91701248 bytes 2024-02-10T01:15:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 200313856 bytes 2024-02-10T01:15:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 223866880 bytes 2024-02-10T01:15:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 269243392 bytes 2024-02-10T01:15:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 285847552 bytes 2024-02-10T01:15:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 347462656 bytes 2024-02-10T01:15:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 354664448 bytes 2024-02-10T01:15:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 400914432 bytes 2024-02-10T01:15:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 471756800 bytes 2024-02-10T01:15:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 496461824 bytes 2024-02-10T01:15:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 527857664 bytes 2024-02-10T01:15:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 567547904 bytes 2024-02-10T01:15:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 643298304 bytes 2024-02-10T01:15:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 723649536 bytes 2024-02-10T01:15:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 762674176 bytes 2024-02-10T01:15:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 878286848 bytes 2024-02-10T01:15:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 937274368 bytes 2024-02-10T01:15:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 966736896 bytes 2024-02-10T01:15:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1083412480 bytes 2024-02-10T01:15:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1129748480 bytes 2024-02-10T01:15:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1183333376 bytes 2024-02-10T01:15:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1246940160 bytes 2024-02-10T01:15:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1268679680 bytes 2024-02-10T01:15:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1275667456 bytes 2024-02-10T01:15:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1296855040 bytes 2024-02-10T01:15:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1318996992 bytes 2024-02-10T01:15:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1360276480 bytes 2024-02-10T01:15:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1400028160 bytes 2024-02-10T01:15:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1448239104 bytes 2024-02-10T01:15:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1481077760 bytes 2024-02-10T01:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1517969408 bytes 2024-02-10T01:15:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1539868672 bytes 2024-02-10T01:15:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1564371968 bytes 2024-02-10T01:15:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1590054912 bytes 2024-02-10T01:15:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1630128128 bytes 2024-02-10T01:15:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1642394624 bytes 2024-02-10T01:15:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1703519232 bytes 2024-02-10T01:15:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1794023424 bytes 2024-02-10T01:15:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1795148800 bytes 2024-02-10T01:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1840013312 bytes 2024-02-10T01:15:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1904259072 bytes 2024-02-10T01:15:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1935958016 bytes 2024-02-10T01:15:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1941840896 bytes 2024-02-10T01:15:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1967510528 bytes 2024-02-10T01:15:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2022282240 bytes 2024-02-10T01:15:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2104948736 bytes 2024-02-10T01:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2118371328 bytes 2024-02-10T01:15:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2143243264 bytes 2024-02-10T01:15:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2180972544 bytes 2024-02-10T01:15:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2244069376 bytes 2024-02-10T01:15:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2267300864 bytes 2024-02-10T01:15:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2328747008 bytes 2024-02-10T01:15:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2394110976 bytes 2024-02-10T01:15:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2454653952 bytes 2024-02-10T01:15:45.495 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9460889024585989) > weight (0.25) 2024-02-10T01:15:45.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9799321206469459) > weight (0.25) 2024-02-10T01:15:45.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6967378888857441) > weight (0.25) 2024-02-10T01:15:45.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9671456462298459) > weight (0.25) 2024-02-10T01:15:45.496 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-02-10T01:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2522938368 bytes 2024-02-10T01:15:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2564017152 bytes 2024-02-10T01:15:45.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2564635648 bytes 2024-02-10T01:15:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2592878592 bytes 2024-02-10T01:15:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2608340992 bytes 2024-02-10T01:15:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2652055552 bytes 2024-02-10T01:15:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2702825472 bytes 2024-02-10T01:15:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2793609216 bytes 2024-02-10T01:15:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2846704640 bytes 2024-02-10T01:15:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2856614912 bytes 2024-02-10T01:15:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2889064448 bytes 2024-02-10T01:15:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2914957312 bytes 2024-02-10T01:15:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2957069312 bytes 2024-02-10T01:15:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2979536896 bytes 2024-02-10T01:15:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3041939456 bytes 2024-02-10T01:15:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3076187136 bytes 2024-02-10T01:15:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3118676992 bytes 2024-02-10T01:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3156450304 bytes 2024-02-10T01:15:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3200588800 bytes 2024-02-10T01:15:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3242261504 bytes 2024-02-10T01:15:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3260417024 bytes 2024-02-10T01:15:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3303402496 bytes 2024-02-10T01:15:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3333362688 bytes 2024-02-10T01:15:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3376322560 bytes 2024-02-10T01:15:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3386187776 bytes 2024-02-10T01:15:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3435932672 bytes 2024-02-10T01:15:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3456884736 bytes 2024-02-10T01:15:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3524147200 bytes 2024-02-10T01:15:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3579054080 bytes 2024-02-10T01:15:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3638424576 bytes 2024-02-10T01:15:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3734123520 bytes 2024-02-10T01:15:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3780760576 bytes 2024-02-10T01:15:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3850193920 bytes 2024-02-10T01:15:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3919450112 bytes 2024-02-10T01:15:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3946209280 bytes 2024-02-10T01:15:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3979245568 bytes 2024-02-10T01:15:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 4018153472 bytes 2024-02-10T01:15:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 4059432960 bytes 2024-02-10T01:15:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 4100184064 bytes 2024-02-10T01:15:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 4130491392 bytes 2024-02-10T01:15:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 4131094528 bytes 2024-02-10T01:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4200990720 bytes 2024-02-10T01:15:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4249724928 bytes 2024-02-10T01:15:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4312155136 bytes 2024-02-10T01:15:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4344132608 bytes 2024-02-10T01:15:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4385402880 bytes 2024-02-10T01:15:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:15:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_13 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:15:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:15:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_13 128M 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2024-02-10T01:15:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:15:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-02-10T01:15:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:15:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-02-10T01:15:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_13': 2024-02-10T01:15:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:15:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:15:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:15:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1307ae087ad 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1307ae087ad 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:51 2024 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:51 2024 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:51 2024 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:15:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:15:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:15:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_13 2024-02-10T01:15:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:15:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:15:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:15:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:15:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:15:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:15:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:15:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 92490752 bytes 2024-02-10T01:15:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 130652160 bytes 2024-02-10T01:15:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 154865664 bytes 2024-02-10T01:15:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 172693504 bytes 2024-02-10T01:15:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 186959872 bytes 2024-02-10T01:15:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 211367936 bytes 2024-02-10T01:15:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 245707776 bytes 2024-02-10T01:15:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 334799872 bytes 2024-02-10T01:15:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 412948480 bytes 2024-02-10T01:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 451668992 bytes 2024-02-10T01:15:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 473176064 bytes 2024-02-10T01:15:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 596540416 bytes 2024-02-10T01:15:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 645472256 bytes 2024-02-10T01:15:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 672183296 bytes 2024-02-10T01:15:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 701461504 bytes 2024-02-10T01:15:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 737564672 bytes 2024-02-10T01:15:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 801494016 bytes 2024-02-10T01:15:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 814785536 bytes 2024-02-10T01:15:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 897763328 bytes 2024-02-10T01:15:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 965151744 bytes 2024-02-10T01:15:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 965769216 bytes 2024-02-10T01:15:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1011927040 bytes 2024-02-10T01:15:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1078873088 bytes 2024-02-10T01:15:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1120098304 bytes 2024-02-10T01:15:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1188041728 bytes 2024-02-10T01:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1232681984 bytes 2024-02-10T01:15:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1264191488 bytes 2024-02-10T01:15:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1319195648 bytes 2024-02-10T01:15:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1379563520 bytes 2024-02-10T01:15:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1380921344 bytes 2024-02-10T01:15:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1474863104 bytes 2024-02-10T01:15:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1506535424 bytes 2024-02-10T01:15:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1528429568 bytes 2024-02-10T01:15:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1565196288 bytes 2024-02-10T01:15:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1610919936 bytes 2024-02-10T01:15:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1636514816 bytes 2024-02-10T01:15:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1659367424 bytes 2024-02-10T01:15:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1692578816 bytes 2024-02-10T01:15:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1736591360 bytes 2024-02-10T01:15:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1805688832 bytes 2024-02-10T01:15:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1841393664 bytes 2024-02-10T01:15:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1844952064 bytes 2024-02-10T01:15:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1882103808 bytes 2024-02-10T01:15:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1977162752 bytes 2024-02-10T01:15:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2045108224 bytes 2024-02-10T01:15:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2085827584 bytes 2024-02-10T01:15:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2128343040 bytes 2024-02-10T01:15:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2255282176 bytes 2024-02-10T01:15:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2283342848 bytes 2024-02-10T01:15:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2362845184 bytes 2024-02-10T01:15:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2383270912 bytes 2024-02-10T01:15:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2400006144 bytes 2024-02-10T01:15:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2414808064 bytes 2024-02-10T01:15:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2421767168 bytes 2024-02-10T01:15:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2432943104 bytes 2024-02-10T01:15:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2449204224 bytes 2024-02-10T01:15:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2483927040 bytes 2024-02-10T01:15:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2554506240 bytes 2024-02-10T01:15:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2591742976 bytes 2024-02-10T01:15:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2602080256 bytes 2024-02-10T01:15:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2627408896 bytes 2024-02-10T01:15:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2633284608 bytes 2024-02-10T01:16:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2655181824 bytes 2024-02-10T01:16:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2700975104 bytes 2024-02-10T01:16:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2721349632 bytes 2024-02-10T01:16:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2722493440 bytes 2024-02-10T01:16:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2771229696 bytes 2024-02-10T01:16:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2806900736 bytes 2024-02-10T01:16:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2839811072 bytes 2024-02-10T01:16:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2935615488 bytes 2024-02-10T01:16:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2947283968 bytes 2024-02-10T01:16:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2990667776 bytes 2024-02-10T01:16:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3015185408 bytes 2024-02-10T01:16:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3122881536 bytes 2024-02-10T01:16:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3180521472 bytes 2024-02-10T01:16:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3200321536 bytes 2024-02-10T01:16:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3257477120 bytes 2024-02-10T01:16:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3335936000 bytes 2024-02-10T01:16:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3372259328 bytes 2024-02-10T01:16:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3463301120 bytes 2024-02-10T01:16:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3490252800 bytes 2024-02-10T01:16:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3559913472 bytes 2024-02-10T01:16:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3606610944 bytes 2024-02-10T01:16:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3661772800 bytes 2024-02-10T01:16:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3670336512 bytes 2024-02-10T01:16:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3715419136 bytes 2024-02-10T01:16:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3729806336 bytes 2024-02-10T01:16:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3763350528 bytes 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_14 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_14 128M 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:16:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2024-02-10T01:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-02-10T01:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:16:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-02-10T01:16:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:16:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_14': 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13138ff3a499 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13138ff3a499 2024-02-10T01:16:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:03 2024 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:03 2024 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:03 2024 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:16:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:16:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:16:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_14 2024-02-10T01:16:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:16:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:16:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:16:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:16:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:16:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 65517568 bytes 2024-02-10T01:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 101960704 bytes 2024-02-10T01:16:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 205769728 bytes 2024-02-10T01:16:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 227837952 bytes 2024-02-10T01:16:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 254420992 bytes 2024-02-10T01:16:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 321579008 bytes 2024-02-10T01:16:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 397916160 bytes 2024-02-10T01:16:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 398504960 bytes 2024-02-10T01:16:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 462030848 bytes 2024-02-10T01:16:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 514129920 bytes 2024-02-10T01:16:05.648 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:16:05.647+0000 7fe71fd26640 -1 reset not still connected to 0x560a41b62c30 2024-02-10T01:16:05.648 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:16:05.647+0000 7fe71fd26640 -1 reset not still connected to 0x560a41b63040 2024-02-10T01:16:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 519000064 bytes 2024-02-10T01:16:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 535353344 bytes 2024-02-10T01:16:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 539783168 bytes 2024-02-10T01:16:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 579570688 bytes 2024-02-10T01:16:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 594408448 bytes 2024-02-10T01:16:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 636347392 bytes 2024-02-10T01:16:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 680891392 bytes 2024-02-10T01:16:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 699745280 bytes 2024-02-10T01:16:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 785029120 bytes 2024-02-10T01:16:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 785664000 bytes 2024-02-10T01:16:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 790809600 bytes 2024-02-10T01:16:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 833549312 bytes 2024-02-10T01:16:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 920902656 bytes 2024-02-10T01:16:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 981156864 bytes 2024-02-10T01:16:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1018933248 bytes 2024-02-10T01:16:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1051918336 bytes 2024-02-10T01:16:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1092089856 bytes 2024-02-10T01:16:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1173729280 bytes 2024-02-10T01:16:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1193678848 bytes 2024-02-10T01:16:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1316282368 bytes 2024-02-10T01:16:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1367193600 bytes 2024-02-10T01:16:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1383076864 bytes 2024-02-10T01:16:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1483999232 bytes 2024-02-10T01:16:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1509052416 bytes 2024-02-10T01:16:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1509668864 bytes 2024-02-10T01:16:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1609400320 bytes 2024-02-10T01:16:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1636106240 bytes 2024-02-10T01:16:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1668080640 bytes 2024-02-10T01:16:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1699309568 bytes 2024-02-10T01:16:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1769372672 bytes 2024-02-10T01:16:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1829198848 bytes 2024-02-10T01:16:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1919299584 bytes 2024-02-10T01:16:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1939227648 bytes 2024-02-10T01:16:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1978306560 bytes 2024-02-10T01:16:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2005325824 bytes 2024-02-10T01:16:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2104299520 bytes 2024-02-10T01:16:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2130787328 bytes 2024-02-10T01:16:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2189136896 bytes 2024-02-10T01:16:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2247378944 bytes 2024-02-10T01:16:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2269155328 bytes 2024-02-10T01:16:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2306714624 bytes 2024-02-10T01:16:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2419295232 bytes 2024-02-10T01:16:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2516882432 bytes 2024-02-10T01:16:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2573260800 bytes 2024-02-10T01:16:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2585800704 bytes 2024-02-10T01:16:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2633716736 bytes 2024-02-10T01:16:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2653868032 bytes 2024-02-10T01:16:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2716783616 bytes 2024-02-10T01:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2756066304 bytes 2024-02-10T01:16:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2772553728 bytes 2024-02-10T01:16:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2800770048 bytes 2024-02-10T01:16:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2877396992 bytes 2024-02-10T01:16:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2946321408 bytes 2024-02-10T01:16:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 3005981696 bytes 2024-02-10T01:16:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 3091920896 bytes 2024-02-10T01:16:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3150274560 bytes 2024-02-10T01:16:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3194674176 bytes 2024-02-10T01:16:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3214981120 bytes 2024-02-10T01:16:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3251873792 bytes 2024-02-10T01:16:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3277393920 bytes 2024-02-10T01:16:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3328602112 bytes 2024-02-10T01:16:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3355215872 bytes 2024-02-10T01:16:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3396076544 bytes 2024-02-10T01:16:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3411031040 bytes 2024-02-10T01:16:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3438936064 bytes 2024-02-10T01:16:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3473501184 bytes 2024-02-10T01:16:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3498780672 bytes 2024-02-10T01:16:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3583941632 bytes 2024-02-10T01:16:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3612806144 bytes 2024-02-10T01:16:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3650537472 bytes 2024-02-10T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3735076864 bytes 2024-02-10T01:16:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:16:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:16:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:16:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_15 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_15 128M 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-02-10T01:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:16:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-02-10T01:16:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_15': 2024-02-10T01:16:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:16:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 131f27c207d5 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.131f27c207d5 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:15 2024 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:15 2024 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:15 2024 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:16:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:16:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:16:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_15 2024-02-10T01:16:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:16:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:16:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:16:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:16:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:16:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 30376960 bytes 2024-02-10T01:16:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 43875328 bytes 2024-02-10T01:16:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 48873472 bytes 2024-02-10T01:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 80423936 bytes 2024-02-10T01:16:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 111260672 bytes 2024-02-10T01:16:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 132067328 bytes 2024-02-10T01:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 167813120 bytes 2024-02-10T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 209174528 bytes 2024-02-10T01:16:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 299177984 bytes 2024-02-10T01:16:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 316968960 bytes 2024-02-10T01:16:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 344681472 bytes 2024-02-10T01:16:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 371629056 bytes 2024-02-10T01:16:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 482010112 bytes 2024-02-10T01:16:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 499621888 bytes 2024-02-10T01:16:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 521199616 bytes 2024-02-10T01:16:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 571464704 bytes 2024-02-10T01:16:18.214 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:16:18.213+0000 7fe71fd26640 -1 reset not still connected to 0x560a46981d40 2024-02-10T01:16:18.214 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:16:18.213+0000 7fe71fd26640 -1 reset not still connected to 0x560a59731930 2024-02-10T01:16:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 602770432 bytes 2024-02-10T01:16:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 684959744 bytes 2024-02-10T01:16:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 739161088 bytes 2024-02-10T01:16:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 756421632 bytes 2024-02-10T01:16:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 787787776 bytes 2024-02-10T01:16:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 807053312 bytes 2024-02-10T01:16:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 849343488 bytes 2024-02-10T01:16:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 882594816 bytes 2024-02-10T01:16:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 945291264 bytes 2024-02-10T01:16:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1033569280 bytes 2024-02-10T01:16:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1065076736 bytes 2024-02-10T01:16:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1139717120 bytes 2024-02-10T01:16:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1156350976 bytes 2024-02-10T01:16:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1208740864 bytes 2024-02-10T01:16:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1231724544 bytes 2024-02-10T01:16:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1272026112 bytes 2024-02-10T01:16:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1276390400 bytes 2024-02-10T01:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1334663168 bytes 2024-02-10T01:16:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1398709248 bytes 2024-02-10T01:16:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1432879104 bytes 2024-02-10T01:16:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1513176064 bytes 2024-02-10T01:16:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1548939264 bytes 2024-02-10T01:16:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1621241856 bytes 2024-02-10T01:16:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1670730752 bytes 2024-02-10T01:16:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1745012736 bytes 2024-02-10T01:16:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1786394624 bytes 2024-02-10T01:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1796039680 bytes 2024-02-10T01:16:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1816058880 bytes 2024-02-10T01:16:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1855876096 bytes 2024-02-10T01:16:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1903549440 bytes 2024-02-10T01:16:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1945945088 bytes 2024-02-10T01:16:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1967654912 bytes 2024-02-10T01:16:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2031744000 bytes 2024-02-10T01:16:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2045952000 bytes 2024-02-10T01:16:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2078166016 bytes 2024-02-10T01:16:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2116905984 bytes 2024-02-10T01:16:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2138566656 bytes 2024-02-10T01:16:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2151071744 bytes 2024-02-10T01:16:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2199018496 bytes 2024-02-10T01:16:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2224200704 bytes 2024-02-10T01:16:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2259930112 bytes 2024-02-10T01:16:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2308292608 bytes 2024-02-10T01:16:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2358548480 bytes 2024-02-10T01:16:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2434420736 bytes 2024-02-10T01:16:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2463850496 bytes 2024-02-10T01:16:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2492236800 bytes 2024-02-10T01:16:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2540063744 bytes 2024-02-10T01:16:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2610409472 bytes 2024-02-10T01:16:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2644512768 bytes 2024-02-10T01:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2682381312 bytes 2024-02-10T01:16:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2725007360 bytes 2024-02-10T01:16:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2775849984 bytes 2024-02-10T01:16:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2845815808 bytes 2024-02-10T01:16:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2858809344 bytes 2024-02-10T01:16:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2874375168 bytes 2024-02-10T01:16:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2946114560 bytes 2024-02-10T01:16:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3039532032 bytes 2024-02-10T01:16:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3095007232 bytes 2024-02-10T01:16:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3114860544 bytes 2024-02-10T01:16:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3143409664 bytes 2024-02-10T01:16:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3172561920 bytes 2024-02-10T01:16:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3281092608 bytes 2024-02-10T01:16:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3299677184 bytes 2024-02-10T01:16:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3354507264 bytes 2024-02-10T01:16:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3427141632 bytes 2024-02-10T01:16:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3488655360 bytes 2024-02-10T01:16:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3566735360 bytes 2024-02-10T01:16:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3598113792 bytes 2024-02-10T01:16:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_16 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:16:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_16 128M 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:16:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:16:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2024-02-10T01:16:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:16:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:16:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-02-10T01:16:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:16:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-02-10T01:16:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:16:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-02-10T01:16:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_16': 2024-02-10T01:16:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:16:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:16:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:16:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 132bdec6648b 2024-02-10T01:16:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.132bdec6648b 2024-02-10T01:16:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:27 2024 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:27 2024 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:27 2024 2024-02-10T01:16:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:16:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:16:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:16:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:16:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:16:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_16 2024-02-10T01:16:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:16:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:16:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:16:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:16:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:16:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 40452096 bytes 2024-02-10T01:16:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 71670784 bytes 2024-02-10T01:16:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 121055232 bytes 2024-02-10T01:16:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 178496512 bytes 2024-02-10T01:16:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 270927872 bytes 2024-02-10T01:16:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 317854720 bytes 2024-02-10T01:16:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 367770624 bytes 2024-02-10T01:16:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 390439936 bytes 2024-02-10T01:16:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 440776704 bytes 2024-02-10T01:16:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 454300672 bytes 2024-02-10T01:16:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 504777728 bytes 2024-02-10T01:16:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 544798720 bytes 2024-02-10T01:16:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 610334720 bytes 2024-02-10T01:16:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 636477440 bytes 2024-02-10T01:16:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 712024064 bytes 2024-02-10T01:16:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 759288832 bytes 2024-02-10T01:16:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 843104256 bytes 2024-02-10T01:16:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 875299840 bytes 2024-02-10T01:16:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 902753280 bytes 2024-02-10T01:16:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 956813312 bytes 2024-02-10T01:16:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1027154944 bytes 2024-02-10T01:16:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1046075392 bytes 2024-02-10T01:16:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1084426240 bytes 2024-02-10T01:16:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1146101760 bytes 2024-02-10T01:16:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1211825152 bytes 2024-02-10T01:16:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1254998016 bytes 2024-02-10T01:16:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1279603712 bytes 2024-02-10T01:16:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1352937472 bytes 2024-02-10T01:16:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1397822464 bytes 2024-02-10T01:16:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1440034816 bytes 2024-02-10T01:16:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1492254720 bytes 2024-02-10T01:16:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1529592832 bytes 2024-02-10T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1557794816 bytes 2024-02-10T01:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1591839744 bytes 2024-02-10T01:16:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1624582144 bytes 2024-02-10T01:16:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1692558336 bytes 2024-02-10T01:16:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1801430016 bytes 2024-02-10T01:16:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1802048512 bytes 2024-02-10T01:16:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1802682368 bytes 2024-02-10T01:16:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1803305984 bytes 2024-02-10T01:16:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1836124160 bytes 2024-02-10T01:16:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1855930368 bytes 2024-02-10T01:16:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1856537600 bytes 2024-02-10T01:16:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1898251264 bytes 2024-02-10T01:16:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1952421888 bytes 2024-02-10T01:16:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1999603712 bytes 2024-02-10T01:16:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2092826624 bytes 2024-02-10T01:16:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2104274944 bytes 2024-02-10T01:16:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2129424384 bytes 2024-02-10T01:16:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2167293952 bytes 2024-02-10T01:16:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2209126400 bytes 2024-02-10T01:16:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2278037504 bytes 2024-02-10T01:16:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2331283456 bytes 2024-02-10T01:16:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2362878976 bytes 2024-02-10T01:16:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2402440192 bytes 2024-02-10T01:16:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2424570880 bytes 2024-02-10T01:16:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2450387968 bytes 2024-02-10T01:16:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2472163328 bytes 2024-02-10T01:16:36.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2513278976 bytes 2024-02-10T01:16:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2536724480 bytes 2024-02-10T01:16:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:16:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:16:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:16:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:16:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_17 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_17 128M 2024-02-10T01:16:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2024-02-10T01:16:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:16:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-02-10T01:16:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:16:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-02-10T01:16:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_17': 2024-02-10T01:16:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1337570a641e 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1337570a641e 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:36 2024 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:36 2024 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:36 2024 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:16:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:16:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:16:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:16:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:16:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_17 2024-02-10T01:16:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:16:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:16:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=9 2024-02-10T01:16:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:16:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:16:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 52132864 bytes 2024-02-10T01:16:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 91822080 bytes 2024-02-10T01:16:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 179311616 bytes 2024-02-10T01:16:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 206407680 bytes 2024-02-10T01:16:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 239833088 bytes 2024-02-10T01:16:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 264190976 bytes 2024-02-10T01:16:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 271495168 bytes 2024-02-10T01:16:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 294603776 bytes 2024-02-10T01:16:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 320658432 bytes 2024-02-10T01:16:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 375306240 bytes 2024-02-10T01:16:38.506 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.39788089014126404) > weight (0.25) 2024-02-10T01:16:38.507 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4631241340203983) > weight (0.25) 2024-02-10T01:16:38.507 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8136638293795881) > weight (0.25) 2024-02-10T01:16:38.507 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7758380909378744) > weight (0.25) 2024-02-10T01:16:38.507 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-02-10T01:16:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 416092160 bytes 2024-02-10T01:16:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 437050368 bytes 2024-02-10T01:16:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 442033152 bytes 2024-02-10T01:16:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 503119872 bytes 2024-02-10T01:16:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 551378944 bytes 2024-02-10T01:16:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 648113152 bytes 2024-02-10T01:16:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 664614912 bytes 2024-02-10T01:16:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 716887040 bytes 2024-02-10T01:16:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 759497728 bytes 2024-02-10T01:16:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 814863360 bytes 2024-02-10T01:16:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 868726784 bytes 2024-02-10T01:16:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 900232192 bytes 2024-02-10T01:16:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 900848640 bytes 2024-02-10T01:16:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 942144512 bytes 2024-02-10T01:16:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1036770304 bytes 2024-02-10T01:16:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1152748544 bytes 2024-02-10T01:16:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1203274752 bytes 2024-02-10T01:16:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1242475520 bytes 2024-02-10T01:16:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1288307712 bytes 2024-02-10T01:16:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1321713664 bytes 2024-02-10T01:16:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1394584576 bytes 2024-02-10T01:16:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1396577280 bytes 2024-02-10T01:16:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1397168128 bytes 2024-02-10T01:16:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1435025408 bytes 2024-02-10T01:16:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1458923520 bytes 2024-02-10T01:16:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1487098880 bytes 2024-02-10T01:16:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1510945792 bytes 2024-02-10T01:16:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1585212416 bytes 2024-02-10T01:16:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1600135168 bytes 2024-02-10T01:16:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1654173696 bytes 2024-02-10T01:16:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1664671744 bytes 2024-02-10T01:16:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1684846592 bytes 2024-02-10T01:16:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1724883968 bytes 2024-02-10T01:16:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1789075456 bytes 2024-02-10T01:16:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1855233024 bytes 2024-02-10T01:16:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1865701376 bytes 2024-02-10T01:16:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1938708480 bytes 2024-02-10T01:16:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1958496256 bytes 2024-02-10T01:16:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2004627456 bytes 2024-02-10T01:16:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2021924864 bytes 2024-02-10T01:16:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2088865792 bytes 2024-02-10T01:16:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2159607808 bytes 2024-02-10T01:16:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2240314368 bytes 2024-02-10T01:16:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2264688640 bytes 2024-02-10T01:16:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2334668800 bytes 2024-02-10T01:16:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2389051392 bytes 2024-02-10T01:16:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2425361408 bytes 2024-02-10T01:16:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2483429376 bytes 2024-02-10T01:16:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2511994880 bytes 2024-02-10T01:16:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2548435968 bytes 2024-02-10T01:16:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2604632064 bytes 2024-02-10T01:16:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2626664448 bytes 2024-02-10T01:16:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2700098560 bytes 2024-02-10T01:16:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2741280768 bytes 2024-02-10T01:16:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2766560256 bytes 2024-02-10T01:16:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2781129728 bytes 2024-02-10T01:16:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2838581248 bytes 2024-02-10T01:16:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2894375936 bytes 2024-02-10T01:16:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2935793664 bytes 2024-02-10T01:16:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2973102080 bytes 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_18 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:16:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_18 128M 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:16:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-02-10T01:16:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:16:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-02-10T01:16:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_18': 2024-02-10T01:16:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:16:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1343ea542103 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1343ea542103 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:46 2024 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:46 2024 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:46 2024 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:16:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:16:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:16:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_18 2024-02-10T01:16:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:16:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:16:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=6 2024-02-10T01:16:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:16:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:16:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 40980480 bytes 2024-02-10T01:16:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 78991360 bytes 2024-02-10T01:16:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 107202560 bytes 2024-02-10T01:16:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 159830016 bytes 2024-02-10T01:16:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 180431872 bytes 2024-02-10T01:16:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 200939520 bytes 2024-02-10T01:16:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 213280768 bytes 2024-02-10T01:16:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 283397120 bytes 2024-02-10T01:16:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 328715264 bytes 2024-02-10T01:16:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 330110976 bytes 2024-02-10T01:16:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 382084096 bytes 2024-02-10T01:16:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 454155264 bytes 2024-02-10T01:16:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 474260480 bytes 2024-02-10T01:16:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 500876288 bytes 2024-02-10T01:16:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 512115712 bytes 2024-02-10T01:16:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 535479296 bytes 2024-02-10T01:16:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 623037440 bytes 2024-02-10T01:16:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 623637504 bytes 2024-02-10T01:16:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 659674112 bytes 2024-02-10T01:16:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 698935296 bytes 2024-02-10T01:16:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 790081536 bytes 2024-02-10T01:16:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 831162368 bytes 2024-02-10T01:16:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 831781888 bytes 2024-02-10T01:16:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 867985408 bytes 2024-02-10T01:16:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 896999424 bytes 2024-02-10T01:16:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 937650176 bytes 2024-02-10T01:16:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 987491328 bytes 2024-02-10T01:16:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1017666560 bytes 2024-02-10T01:16:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1030183936 bytes 2024-02-10T01:16:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1064351744 bytes 2024-02-10T01:16:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1136604160 bytes 2024-02-10T01:16:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1174335488 bytes 2024-02-10T01:16:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1196272640 bytes 2024-02-10T01:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1244120064 bytes 2024-02-10T01:16:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1264133120 bytes 2024-02-10T01:16:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1321686016 bytes 2024-02-10T01:16:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1339553792 bytes 2024-02-10T01:16:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1435160576 bytes 2024-02-10T01:16:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1479293952 bytes 2024-02-10T01:16:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1527771136 bytes 2024-02-10T01:16:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1563884544 bytes 2024-02-10T01:16:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1634359296 bytes 2024-02-10T01:16:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1693102080 bytes 2024-02-10T01:16:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1696811008 bytes 2024-02-10T01:16:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1777113088 bytes 2024-02-10T01:16:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1803472896 bytes 2024-02-10T01:16:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1822552064 bytes 2024-02-10T01:16:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1877397504 bytes 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_19 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-02-10T01:16:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_19 128M 2024-02-10T01:16:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:16:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2024-02-10T01:16:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:16:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-02-10T01:16:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:16:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-02-10T01:16:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_19': 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 134f53dc8423 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.134f53dc8423 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:52 2024 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:52 2024 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:52 2024 2024-02-10T01:16:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:16:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:16:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:16:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:16:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:16:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_19 2024-02-10T01:16:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:16:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:16:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:16:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:16:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:16:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:16:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 31295488 bytes 2024-02-10T01:16:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 62059520 bytes 2024-02-10T01:16:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 83507200 bytes 2024-02-10T01:16:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 150509568 bytes 2024-02-10T01:16:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 214080512 bytes 2024-02-10T01:16:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 233780224 bytes 2024-02-10T01:16:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 262840320 bytes 2024-02-10T01:16:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 339213312 bytes 2024-02-10T01:16:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 351659008 bytes 2024-02-10T01:16:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 404294656 bytes 2024-02-10T01:16:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 420709376 bytes 2024-02-10T01:16:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 465729536 bytes 2024-02-10T01:16:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 472322048 bytes 2024-02-10T01:16:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 535676928 bytes 2024-02-10T01:16:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 636154880 bytes 2024-02-10T01:16:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 667993088 bytes 2024-02-10T01:16:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 689503232 bytes 2024-02-10T01:16:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 733486080 bytes 2024-02-10T01:16:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 760697856 bytes 2024-02-10T01:16:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 836146176 bytes 2024-02-10T01:16:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 845253632 bytes 2024-02-10T01:16:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 908503040 bytes 2024-02-10T01:16:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 952338432 bytes 2024-02-10T01:16:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1033686016 bytes 2024-02-10T01:16:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1073750016 bytes 2024-02-10T01:16:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1092849664 bytes 2024-02-10T01:16:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1126811648 bytes 2024-02-10T01:16:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1201468416 bytes 2024-02-10T01:16:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1236752384 bytes 2024-02-10T01:16:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1287424000 bytes 2024-02-10T01:16:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1311587328 bytes 2024-02-10T01:16:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1312189440 bytes 2024-02-10T01:16:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1346532352 bytes 2024-02-10T01:16:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1399984128 bytes 2024-02-10T01:16:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1448247296 bytes 2024-02-10T01:16:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1513290752 bytes 2024-02-10T01:16:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1519036416 bytes 2024-02-10T01:16:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1566152704 bytes 2024-02-10T01:16:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1601298432 bytes 2024-02-10T01:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1653382144 bytes 2024-02-10T01:16:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1678212096 bytes 2024-02-10T01:16:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1716912128 bytes 2024-02-10T01:16:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1797786624 bytes 2024-02-10T01:16:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1822100480 bytes 2024-02-10T01:16:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1886964736 bytes 2024-02-10T01:16:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1907960832 bytes 2024-02-10T01:16:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1969491968 bytes 2024-02-10T01:16:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1982425088 bytes 2024-02-10T01:17:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2008547328 bytes 2024-02-10T01:17:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2064222208 bytes 2024-02-10T01:17:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2122018816 bytes 2024-02-10T01:17:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2126043136 bytes 2024-02-10T01:17:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2182858752 bytes 2024-02-10T01:17:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2195461120 bytes 2024-02-10T01:17:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:17:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_20 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_20 128M 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2024-02-10T01:17:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:17:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-02-10T01:17:01.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:17:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_20': 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 135b22d68b30 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.135b22d68b30 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:17:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:00 2024 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:00 2024 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:00 2024 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:17:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:17:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:17:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_20 2024-02-10T01:17:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:17:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:17:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=10 2024-02-10T01:17:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:17:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:17:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 53385216 bytes 2024-02-10T01:17:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 90982400 bytes 2024-02-10T01:17:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 159874048 bytes 2024-02-10T01:17:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 220807168 bytes 2024-02-10T01:17:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 246413312 bytes 2024-02-10T01:17:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 274997248 bytes 2024-02-10T01:17:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 310995968 bytes 2024-02-10T01:17:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 358420480 bytes 2024-02-10T01:17:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 428583936 bytes 2024-02-10T01:17:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 480736256 bytes 2024-02-10T01:17:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 495727616 bytes 2024-02-10T01:17:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 505671680 bytes 2024-02-10T01:17:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 521844736 bytes 2024-02-10T01:17:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 601260032 bytes 2024-02-10T01:17:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 640428032 bytes 2024-02-10T01:17:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 695915520 bytes 2024-02-10T01:17:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 700544000 bytes 2024-02-10T01:17:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 799559680 bytes 2024-02-10T01:17:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 866797568 bytes 2024-02-10T01:17:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 900370432 bytes 2024-02-10T01:17:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 944006144 bytes 2024-02-10T01:17:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1002283008 bytes 2024-02-10T01:17:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1019292672 bytes 2024-02-10T01:17:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1083557888 bytes 2024-02-10T01:17:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1127766016 bytes 2024-02-10T01:17:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1133168640 bytes 2024-02-10T01:17:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1201779712 bytes 2024-02-10T01:17:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1248326656 bytes 2024-02-10T01:17:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1298833408 bytes 2024-02-10T01:17:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1368701952 bytes 2024-02-10T01:17:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1421135872 bytes 2024-02-10T01:17:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1476464640 bytes 2024-02-10T01:17:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1533120512 bytes 2024-02-10T01:17:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1563776000 bytes 2024-02-10T01:17:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1602727936 bytes 2024-02-10T01:17:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1666296832 bytes 2024-02-10T01:17:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1681977344 bytes 2024-02-10T01:17:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1761195008 bytes 2024-02-10T01:17:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1800198144 bytes 2024-02-10T01:17:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1838614528 bytes 2024-02-10T01:17:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1930699776 bytes 2024-02-10T01:17:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1954499584 bytes 2024-02-10T01:17:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1989183488 bytes 2024-02-10T01:17:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2025665536 bytes 2024-02-10T01:17:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2052738048 bytes 2024-02-10T01:17:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2096221184 bytes 2024-02-10T01:17:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2115576832 bytes 2024-02-10T01:17:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2166949888 bytes 2024-02-10T01:17:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2236984320 bytes 2024-02-10T01:17:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2253216768 bytes 2024-02-10T01:17:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2269404160 bytes 2024-02-10T01:17:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2276665344 bytes 2024-02-10T01:17:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2294037504 bytes 2024-02-10T01:17:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2340541440 bytes 2024-02-10T01:17:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2373184512 bytes 2024-02-10T01:17:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2383209472 bytes 2024-02-10T01:17:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2383811584 bytes 2024-02-10T01:17:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2462902272 bytes 2024-02-10T01:17:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2532665344 bytes 2024-02-10T01:17:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2533248000 bytes 2024-02-10T01:17:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2544979968 bytes 2024-02-10T01:17:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2604747776 bytes 2024-02-10T01:17:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2620664832 bytes 2024-02-10T01:17:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2639093760 bytes 2024-02-10T01:17:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2672712704 bytes 2024-02-10T01:17:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2729275392 bytes 2024-02-10T01:17:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2756469760 bytes 2024-02-10T01:17:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2760108032 bytes 2024-02-10T01:17:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2791975936 bytes 2024-02-10T01:17:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2820170752 bytes 2024-02-10T01:17:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2850552832 bytes 2024-02-10T01:17:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2851163136 bytes 2024-02-10T01:17:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2872051712 bytes 2024-02-10T01:17:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 2890689536 bytes 2024-02-10T01:17:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 2990678016 bytes 2024-02-10T01:17:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3058045952 bytes 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_21 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:17:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:17:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_21 128M 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2024-02-10T01:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-02-10T01:17:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:17:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-02-10T01:17:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_21': 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13676e1c1986 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13676e1c1986 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:17:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:17:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:11 2024 2024-02-10T01:17:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:11 2024 2024-02-10T01:17:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:11 2024 2024-02-10T01:17:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:17:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:17:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:17:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:17:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:17:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_21 2024-02-10T01:17:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:17:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:17:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:17:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:17:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:17:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 54063104 bytes 2024-02-10T01:17:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 142365696 bytes 2024-02-10T01:17:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 217627648 bytes 2024-02-10T01:17:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 293134336 bytes 2024-02-10T01:17:13.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 324040704 bytes 2024-02-10T01:17:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 407942144 bytes 2024-02-10T01:17:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 454387712 bytes 2024-02-10T01:17:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 505473024 bytes 2024-02-10T01:17:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 561620992 bytes 2024-02-10T01:17:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 646899712 bytes 2024-02-10T01:17:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 654019584 bytes 2024-02-10T01:17:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 727579648 bytes 2024-02-10T01:17:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 765799424 bytes 2024-02-10T01:17:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 790382592 bytes 2024-02-10T01:17:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 845478912 bytes 2024-02-10T01:17:14.497 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-02-10T01:17:14.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-02-10T01:17:14.498 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-02-10T01:17:14.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:2024-02-10T01:17:14.498+0000 7f4095078640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 46714) UID: 1000 2024-02-10T01:17:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 889935872 bytes 2024-02-10T01:17:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 947836928 bytes 2024-02-10T01:17:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 985556992 bytes 2024-02-10T01:17:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 1033413632 bytes 2024-02-10T01:17:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1100579840 bytes 2024-02-10T01:17:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1108550656 bytes 2024-02-10T01:17:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1150700544 bytes 2024-02-10T01:17:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1205224448 bytes 2024-02-10T01:17:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1218396160 bytes 2024-02-10T01:17:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1272779776 bytes 2024-02-10T01:17:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1308165120 bytes 2024-02-10T01:17:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1372604416 bytes 2024-02-10T01:17:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1382292480 bytes 2024-02-10T01:17:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1417197568 bytes 2024-02-10T01:17:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1437282304 bytes 2024-02-10T01:17:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1505640448 bytes 2024-02-10T01:17:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1537894400 bytes 2024-02-10T01:17:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1570030592 bytes 2024-02-10T01:17:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1638318080 bytes 2024-02-10T01:17:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1694464000 bytes 2024-02-10T01:17:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1744231424 bytes 2024-02-10T01:17:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1771282432 bytes 2024-02-10T01:17:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1795239936 bytes 2024-02-10T01:17:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1812285440 bytes 2024-02-10T01:17:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1899343872 bytes 2024-02-10T01:17:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1909223424 bytes 2024-02-10T01:17:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1934245888 bytes 2024-02-10T01:17:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1955356672 bytes 2024-02-10T01:17:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2019731456 bytes 2024-02-10T01:17:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2056109056 bytes 2024-02-10T01:17:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2116487168 bytes 2024-02-10T01:17:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2157600768 bytes 2024-02-10T01:17:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2236269568 bytes 2024-02-10T01:17:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2300262400 bytes 2024-02-10T01:17:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2365741056 bytes 2024-02-10T01:17:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2418232320 bytes 2024-02-10T01:17:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2456656896 bytes 2024-02-10T01:17:19.499 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-02-10T01:17:19.499 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:17:19.499 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-02-10T01:17:19.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-02-10T01:17:19.500 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-02-10T01:17:19.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-02-10T01:17:19.503 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-02-10T01:17:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2500250624 bytes 2024-02-10T01:17:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2529776640 bytes 2024-02-10T01:17:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2589155328 bytes 2024-02-10T01:17:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2666689536 bytes 2024-02-10T01:17:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2717066240 bytes 2024-02-10T01:17:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2730060800 bytes 2024-02-10T01:17:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2772660224 bytes 2024-02-10T01:17:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2808002560 bytes 2024-02-10T01:17:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2877569024 bytes 2024-02-10T01:17:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2933100544 bytes 2024-02-10T01:17:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2999222272 bytes 2024-02-10T01:17:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 3021650944 bytes 2024-02-10T01:17:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 3063690240 bytes 2024-02-10T01:17:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3116235776 bytes 2024-02-10T01:17:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3135896576 bytes 2024-02-10T01:17:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3178839040 bytes 2024-02-10T01:17:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3179418624 bytes 2024-02-10T01:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3225984000 bytes 2024-02-10T01:17:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3249959936 bytes 2024-02-10T01:17:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3298073600 bytes 2024-02-10T01:17:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3298691072 bytes 2024-02-10T01:17:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3365381120 bytes 2024-02-10T01:17:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3399489536 bytes 2024-02-10T01:17:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3451375616 bytes 2024-02-10T01:17:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3498324992 bytes 2024-02-10T01:17:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3560719360 bytes 2024-02-10T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3645221888 bytes 2024-02-10T01:17:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3694980096 bytes 2024-02-10T01:17:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3720508416 bytes 2024-02-10T01:17:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3774849024 bytes 2024-02-10T01:17:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3797408768 bytes 2024-02-10T01:17:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3847247872 bytes 2024-02-10T01:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3907021824 bytes 2024-02-10T01:17:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3946883072 bytes 2024-02-10T01:17:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3980301312 bytes 2024-02-10T01:17:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 4021572608 bytes 2024-02-10T01:17:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 4038936576 bytes 2024-02-10T01:17:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 4086121472 bytes 2024-02-10T01:17:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 4103903232 bytes 2024-02-10T01:17:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 4120068096 bytes 2024-02-10T01:17:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 4169084928 bytes 2024-02-10T01:17:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 4238322688 bytes 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_22 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_22 128M 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2024-02-10T01:17:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:17:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:17:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-02-10T01:17:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:17:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-02-10T01:17:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:17:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-02-10T01:17:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_22': 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 137c85371d87 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.137c85371d87 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:25 2024 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:25 2024 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:25 2024 2024-02-10T01:17:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:17:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:17:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:17:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:17:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:17:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_22 2024-02-10T01:17:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:17:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:17:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=9 2024-02-10T01:17:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:17:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:17:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 33680384 bytes 2024-02-10T01:17:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 62887936 bytes 2024-02-10T01:17:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 91664384 bytes 2024-02-10T01:17:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 137284608 bytes 2024-02-10T01:17:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 165905408 bytes 2024-02-10T01:17:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 166529024 bytes 2024-02-10T01:17:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 190127104 bytes 2024-02-10T01:17:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 265582592 bytes 2024-02-10T01:17:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 304633856 bytes 2024-02-10T01:17:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 352854016 bytes 2024-02-10T01:17:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 406307840 bytes 2024-02-10T01:17:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 453249024 bytes 2024-02-10T01:17:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 501273600 bytes 2024-02-10T01:17:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 541139968 bytes 2024-02-10T01:17:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 582800384 bytes 2024-02-10T01:17:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 639770624 bytes 2024-02-10T01:17:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 650862592 bytes 2024-02-10T01:17:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 673159168 bytes 2024-02-10T01:17:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 712228864 bytes 2024-02-10T01:17:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 733706240 bytes 2024-02-10T01:17:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 741457920 bytes 2024-02-10T01:17:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 747065344 bytes 2024-02-10T01:17:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 805682176 bytes 2024-02-10T01:17:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 861614080 bytes 2024-02-10T01:17:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 875617280 bytes 2024-02-10T01:17:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 895458304 bytes 2024-02-10T01:17:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 898654208 bytes 2024-02-10T01:17:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 902279168 bytes 2024-02-10T01:17:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 944914432 bytes 2024-02-10T01:17:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1029412864 bytes 2024-02-10T01:17:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1089975296 bytes 2024-02-10T01:17:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1110941696 bytes 2024-02-10T01:17:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1194327040 bytes 2024-02-10T01:17:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1218141184 bytes 2024-02-10T01:17:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1218761728 bytes 2024-02-10T01:17:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1250338816 bytes 2024-02-10T01:17:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1286858752 bytes 2024-02-10T01:17:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1308721152 bytes 2024-02-10T01:17:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1354949632 bytes 2024-02-10T01:17:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1393487872 bytes 2024-02-10T01:17:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1441314816 bytes 2024-02-10T01:17:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1457965056 bytes 2024-02-10T01:17:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1485672448 bytes 2024-02-10T01:17:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1525272576 bytes 2024-02-10T01:17:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1525915648 bytes 2024-02-10T01:17:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1562609664 bytes 2024-02-10T01:17:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1596581888 bytes 2024-02-10T01:17:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1677724672 bytes 2024-02-10T01:17:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1716005888 bytes 2024-02-10T01:17:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1755862016 bytes 2024-02-10T01:17:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1832488960 bytes 2024-02-10T01:17:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1854496768 bytes 2024-02-10T01:17:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 1986362368 bytes 2024-02-10T01:17:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2030825472 bytes 2024-02-10T01:17:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2104774656 bytes 2024-02-10T01:17:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2116646912 bytes 2024-02-10T01:17:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2117269504 bytes 2024-02-10T01:17:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2140995584 bytes 2024-02-10T01:17:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2229178368 bytes 2024-02-10T01:17:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2299441152 bytes 2024-02-10T01:17:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2411276288 bytes 2024-02-10T01:17:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2468692992 bytes 2024-02-10T01:17:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2509601792 bytes 2024-02-10T01:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2576197632 bytes 2024-02-10T01:17:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2597989376 bytes 2024-02-10T01:17:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2657590272 bytes 2024-02-10T01:17:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2710441984 bytes 2024-02-10T01:17:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2740178944 bytes 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_23 2024-02-10T01:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_23 128M 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2024-02-10T01:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-02-10T01:17:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:17:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-02-10T01:17:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:17:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_23': 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1388762dcd96 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1388762dcd96 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:17:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:35 2024 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:35 2024 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:35 2024 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:17:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:17:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:17:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_23 2024-02-10T01:17:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:17:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:17:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=12 2024-02-10T01:17:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:17:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:17:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 576512 bytes 2024-02-10T01:17:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 60641280 bytes 2024-02-10T01:17:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 79649792 bytes 2024-02-10T01:17:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 136919040 bytes 2024-02-10T01:17:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 187173888 bytes 2024-02-10T01:17:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 212415488 bytes 2024-02-10T01:17:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 261088256 bytes 2024-02-10T01:17:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 299927552 bytes 2024-02-10T01:17:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 358274048 bytes 2024-02-10T01:17:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 402316288 bytes 2024-02-10T01:17:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 425149440 bytes 2024-02-10T01:17:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 432229376 bytes 2024-02-10T01:17:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 468994048 bytes 2024-02-10T01:17:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 492463104 bytes 2024-02-10T01:17:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 523752448 bytes 2024-02-10T01:17:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 554382336 bytes 2024-02-10T01:17:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 593939456 bytes 2024-02-10T01:17:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 670946304 bytes 2024-02-10T01:17:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 680582144 bytes 2024-02-10T01:17:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 789508096 bytes 2024-02-10T01:17:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 822733824 bytes 2024-02-10T01:17:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 857065472 bytes 2024-02-10T01:17:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 906152960 bytes 2024-02-10T01:17:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 935103488 bytes 2024-02-10T01:17:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 989971456 bytes 2024-02-10T01:17:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1008113664 bytes 2024-02-10T01:17:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1027311616 bytes 2024-02-10T01:17:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1043561472 bytes 2024-02-10T01:17:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1113945088 bytes 2024-02-10T01:17:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1152150528 bytes 2024-02-10T01:17:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1194266624 bytes 2024-02-10T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1234529280 bytes 2024-02-10T01:17:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1314604032 bytes 2024-02-10T01:17:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1357077504 bytes 2024-02-10T01:17:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1436640256 bytes 2024-02-10T01:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1530663936 bytes 2024-02-10T01:17:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1552935936 bytes 2024-02-10T01:17:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1565440000 bytes 2024-02-10T01:17:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1583229952 bytes 2024-02-10T01:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1640360960 bytes 2024-02-10T01:17:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1676654592 bytes 2024-02-10T01:17:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1761282048 bytes 2024-02-10T01:17:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1817013248 bytes 2024-02-10T01:17:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1877536768 bytes 2024-02-10T01:17:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1957680128 bytes 2024-02-10T01:17:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2023770112 bytes 2024-02-10T01:17:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2036938752 bytes 2024-02-10T01:17:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2087013376 bytes 2024-02-10T01:17:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2095915008 bytes 2024-02-10T01:17:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2178734080 bytes 2024-02-10T01:17:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2219798528 bytes 2024-02-10T01:17:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2284680192 bytes 2024-02-10T01:17:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2285316096 bytes 2024-02-10T01:17:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2342414336 bytes 2024-02-10T01:17:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2435284992 bytes 2024-02-10T01:17:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2519763968 bytes 2024-02-10T01:17:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2555149312 bytes 2024-02-10T01:17:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2650604544 bytes 2024-02-10T01:17:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2660389888 bytes 2024-02-10T01:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2679069696 bytes 2024-02-10T01:17:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2743305216 bytes 2024-02-10T01:17:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2799129600 bytes 2024-02-10T01:17:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2841936896 bytes 2024-02-10T01:17:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2872586240 bytes 2024-02-10T01:17:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2903724032 bytes 2024-02-10T01:17:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2961437696 bytes 2024-02-10T01:17:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3027563520 bytes 2024-02-10T01:17:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3060196352 bytes 2024-02-10T01:17:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3065580544 bytes 2024-02-10T01:17:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3084726272 bytes 2024-02-10T01:17:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3111677952 bytes 2024-02-10T01:17:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3112287232 bytes 2024-02-10T01:17:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3145131008 bytes 2024-02-10T01:17:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3282344960 bytes 2024-02-10T01:17:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3355283456 bytes 2024-02-10T01:17:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3426721792 bytes 2024-02-10T01:17:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3434974208 bytes 2024-02-10T01:17:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3441147904 bytes 2024-02-10T01:17:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3501321216 bytes 2024-02-10T01:17:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3528122368 bytes 2024-02-10T01:17:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3586915328 bytes 2024-02-10T01:17:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3619141632 bytes 2024-02-10T01:17:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3684651008 bytes 2024-02-10T01:17:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3685268480 bytes 2024-02-10T01:17:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3774385152 bytes 2024-02-10T01:17:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3785282560 bytes 2024-02-10T01:17:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3814486016 bytes 2024-02-10T01:17:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3841499136 bytes 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_24 2024-02-10T01:17:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:17:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_24 128M 2024-02-10T01:17:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2024-02-10T01:17:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-02-10T01:17:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:17:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_24': 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1394a57b3ad3 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1394a57b3ad3 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:48 2024 2024-02-10T01:17:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:48 2024 2024-02-10T01:17:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:48 2024 2024-02-10T01:17:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:17:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:17:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:17:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:17:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:17:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_24 2024-02-10T01:17:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:17:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:17:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=9 2024-02-10T01:17:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:17:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:17:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 28619776 bytes 2024-02-10T01:17:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 127959040 bytes 2024-02-10T01:17:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 155615232 bytes 2024-02-10T01:17:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 174020608 bytes 2024-02-10T01:17:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 212899840 bytes 2024-02-10T01:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 248414208 bytes 2024-02-10T01:17:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 294141952 bytes 2024-02-10T01:17:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 319546368 bytes 2024-02-10T01:17:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 413430784 bytes 2024-02-10T01:17:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 478751744 bytes 2024-02-10T01:17:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 512066560 bytes 2024-02-10T01:17:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 550118400 bytes 2024-02-10T01:17:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 585089024 bytes 2024-02-10T01:17:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 602757120 bytes 2024-02-10T01:17:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 675968000 bytes 2024-02-10T01:17:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 718375936 bytes 2024-02-10T01:17:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 799081472 bytes 2024-02-10T01:17:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 821712896 bytes 2024-02-10T01:17:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 823614464 bytes 2024-02-10T01:17:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 920429568 bytes 2024-02-10T01:17:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 990272512 bytes 2024-02-10T01:17:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1027623936 bytes 2024-02-10T01:17:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1028200448 bytes 2024-02-10T01:17:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1111912448 bytes 2024-02-10T01:17:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1131209728 bytes 2024-02-10T01:17:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1156032512 bytes 2024-02-10T01:17:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1192966144 bytes 2024-02-10T01:17:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1273811968 bytes 2024-02-10T01:17:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1304142848 bytes 2024-02-10T01:17:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1394508800 bytes 2024-02-10T01:17:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1492879360 bytes 2024-02-10T01:17:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1559397376 bytes 2024-02-10T01:17:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1632432128 bytes 2024-02-10T01:17:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1656895488 bytes 2024-02-10T01:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1684945920 bytes 2024-02-10T01:17:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1692673024 bytes 2024-02-10T01:17:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1722304512 bytes 2024-02-10T01:17:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1777638400 bytes 2024-02-10T01:17:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1783608320 bytes 2024-02-10T01:17:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1808773120 bytes 2024-02-10T01:17:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1838060544 bytes 2024-02-10T01:17:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1955188736 bytes 2024-02-10T01:17:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1965040640 bytes 2024-02-10T01:17:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2035099648 bytes 2024-02-10T01:17:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2086188032 bytes 2024-02-10T01:17:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2090518528 bytes 2024-02-10T01:17:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2148151296 bytes 2024-02-10T01:17:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2167118848 bytes 2024-02-10T01:17:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2234103808 bytes 2024-02-10T01:17:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2323545088 bytes 2024-02-10T01:17:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2381774848 bytes 2024-02-10T01:17:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2426991616 bytes 2024-02-10T01:17:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2449557504 bytes 2024-02-10T01:17:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2464416768 bytes 2024-02-10T01:17:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2495967232 bytes 2024-02-10T01:17:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2501976064 bytes 2024-02-10T01:17:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2542386176 bytes 2024-02-10T01:17:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2555725824 bytes 2024-02-10T01:17:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2572078080 bytes 2024-02-10T01:17:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2598769664 bytes 2024-02-10T01:17:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2675412992 bytes 2024-02-10T01:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2747669504 bytes 2024-02-10T01:17:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2813050880 bytes 2024-02-10T01:17:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2899556352 bytes 2024-02-10T01:17:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2934718464 bytes 2024-02-10T01:17:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:17:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:17:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:17:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_25 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:17:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_25 128M 2024-02-10T01:17:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:17:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2024-02-10T01:17:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:17:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:17:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:17:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-02-10T01:17:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:17:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:17:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-02-10T01:17:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:17:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_25': 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 139f348b77ce 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.139f348b77ce 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:58 2024 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:58 2024 2024-02-10T01:17:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:58 2024 2024-02-10T01:17:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:17:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:17:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:17:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:17:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:17:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_25 2024-02-10T01:17:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:17:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:17:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:17:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:17:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:17:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:17:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:17:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 54930432 bytes 2024-02-10T01:17:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 81708032 bytes 2024-02-10T01:17:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 103387136 bytes 2024-02-10T01:17:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 134526976 bytes 2024-02-10T01:17:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 135142400 bytes 2024-02-10T01:17:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 160315392 bytes 2024-02-10T01:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 160901120 bytes 2024-02-10T01:18:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 190730240 bytes 2024-02-10T01:18:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 234697728 bytes 2024-02-10T01:18:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 260787200 bytes 2024-02-10T01:18:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 306452480 bytes 2024-02-10T01:18:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 358929408 bytes 2024-02-10T01:18:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 388669440 bytes 2024-02-10T01:18:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 413093888 bytes 2024-02-10T01:18:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 456592384 bytes 2024-02-10T01:18:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 471663616 bytes 2024-02-10T01:18:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 501217280 bytes 2024-02-10T01:18:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 514329600 bytes 2024-02-10T01:18:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 558766080 bytes 2024-02-10T01:18:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 563785728 bytes 2024-02-10T01:18:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 592009216 bytes 2024-02-10T01:18:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 615540736 bytes 2024-02-10T01:18:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 658874368 bytes 2024-02-10T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 720679936 bytes 2024-02-10T01:18:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 763403264 bytes 2024-02-10T01:18:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 765476864 bytes 2024-02-10T01:18:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 835878912 bytes 2024-02-10T01:18:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 865846272 bytes 2024-02-10T01:18:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 908008448 bytes 2024-02-10T01:18:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 929778688 bytes 2024-02-10T01:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 994514944 bytes 2024-02-10T01:18:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1032739840 bytes 2024-02-10T01:18:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1096814592 bytes 2024-02-10T01:18:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1128659968 bytes 2024-02-10T01:18:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1157325824 bytes 2024-02-10T01:18:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1192721408 bytes 2024-02-10T01:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1226602496 bytes 2024-02-10T01:18:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1256115200 bytes 2024-02-10T01:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1305413632 bytes 2024-02-10T01:18:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1316690944 bytes 2024-02-10T01:18:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1372172288 bytes 2024-02-10T01:18:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1393267712 bytes 2024-02-10T01:18:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1485175808 bytes 2024-02-10T01:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1492785152 bytes 2024-02-10T01:18:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1504595968 bytes 2024-02-10T01:18:04.988 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:18:04.986+0000 7fe71fd26640 -1 reset not still connected to 0x560a39ed3930 2024-02-10T01:18:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1538430976 bytes 2024-02-10T01:18:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1601289216 bytes 2024-02-10T01:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1668167680 bytes 2024-02-10T01:18:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1677984768 bytes 2024-02-10T01:18:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1709067264 bytes 2024-02-10T01:18:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1744741376 bytes 2024-02-10T01:18:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1780752384 bytes 2024-02-10T01:18:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 1870482432 bytes 2024-02-10T01:18:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 1899622400 bytes 2024-02-10T01:18:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 1938300928 bytes 2024-02-10T01:18:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 1985728512 bytes 2024-02-10T01:18:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2046264320 bytes 2024-02-10T01:18:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2103501824 bytes 2024-02-10T01:18:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2129415168 bytes 2024-02-10T01:18:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2161270784 bytes 2024-02-10T01:18:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2206850048 bytes 2024-02-10T01:18:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2237990912 bytes 2024-02-10T01:18:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_26 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_26 128M 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2024-02-10T01:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:18:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-02-10T01:18:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:18:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-02-10T01:18:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_26': 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13ab6fd0e5ff 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13ab6fd0e5ff 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:07 2024 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:07 2024 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:07 2024 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_26 2024-02-10T01:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:18:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:18:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:18:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:18:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 623616 bytes 2024-02-10T01:18:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 51536896 bytes 2024-02-10T01:18:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 98537472 bytes 2024-02-10T01:18:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 130029568 bytes 2024-02-10T01:18:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 154884096 bytes 2024-02-10T01:18:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 189741056 bytes 2024-02-10T01:18:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 191142912 bytes 2024-02-10T01:18:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 238790656 bytes 2024-02-10T01:18:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 259587072 bytes 2024-02-10T01:18:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 293604352 bytes 2024-02-10T01:18:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 325072896 bytes 2024-02-10T01:18:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 325681152 bytes 2024-02-10T01:18:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 358074368 bytes 2024-02-10T01:18:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 390568960 bytes 2024-02-10T01:18:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 420268032 bytes 2024-02-10T01:18:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 485485568 bytes 2024-02-10T01:18:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 526275584 bytes 2024-02-10T01:18:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 616459264 bytes 2024-02-10T01:18:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 666873856 bytes 2024-02-10T01:18:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 702237696 bytes 2024-02-10T01:18:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 717382656 bytes 2024-02-10T01:18:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 782811136 bytes 2024-02-10T01:18:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 792442880 bytes 2024-02-10T01:18:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 802283520 bytes 2024-02-10T01:18:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 829815808 bytes 2024-02-10T01:18:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 857406464 bytes 2024-02-10T01:18:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 872193024 bytes 2024-02-10T01:18:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 921755648 bytes 2024-02-10T01:18:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 990642176 bytes 2024-02-10T01:18:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1066711040 bytes 2024-02-10T01:18:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1120372736 bytes 2024-02-10T01:18:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1163764736 bytes 2024-02-10T01:18:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1164365824 bytes 2024-02-10T01:18:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1189563392 bytes 2024-02-10T01:18:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1190201344 bytes 2024-02-10T01:18:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1233434624 bytes 2024-02-10T01:18:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1279278080 bytes 2024-02-10T01:18:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1309627392 bytes 2024-02-10T01:18:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1321390080 bytes 2024-02-10T01:18:12.507 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7085337992987722) > weight (0.25) 2024-02-10T01:18:12.507 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-02-10T01:18:12.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-02-10T01:18:12.508 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-02-10T01:18:12.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi060.stderr:2024-02-10T01:18:12.508+0000 7f5383038640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 44970) UID: 1000 2024-02-10T01:18:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1365428224 bytes 2024-02-10T01:18:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1387170816 bytes 2024-02-10T01:18:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1403682816 bytes 2024-02-10T01:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1458466816 bytes 2024-02-10T01:18:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1479981056 bytes 2024-02-10T01:18:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1587830784 bytes 2024-02-10T01:18:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1597541376 bytes 2024-02-10T01:18:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1650276352 bytes 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff497c2b640 -1 rbd::mirror::ImageReplayer: 0x55c392258500 [3/97797f84-42fa-4648-a6af-57a4b97a77a2] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff49842c640 -1 rbd::mirror::ImageReplayer: 0x55c392258500 [3/97797f84-42fa-4648-a6af-57a4b97a77a2] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff497c2b640 -1 rbd::mirror::ImageReplayer: 0x55c392258a00 [3/278e9f1e-33e4-4605-97dc-7d7e289de2fd] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff49842c640 -1 rbd::mirror::ImageReplayer: 0x55c392258a00 [3/278e9f1e-33e4-4605-97dc-7d7e289de2fd] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff48b412640 -1 rbd::mirror::ImageReplayer: 0x55c392258a00 [3/278e9f1e-33e4-4605-97dc-7d7e289de2fd] start: already running 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff497c2b640 -1 rbd::mirror::ImageReplayer: 0x55c392258280 [3/69e299aa-5815-4202-b2c2-9fd34a982ad9] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.646+0000 7ff49842c640 -1 rbd::mirror::ImageReplayer: 0x55c392258280 [3/69e299aa-5815-4202-b2c2-9fd34a982ad9] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff7cfc8640 -1 rbd::mirror::ImageReplayer: 0x56401823ca00 [3/4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1] start: already running 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823c500 [3/c893d42f-80cd-48db-be84-9a6dfee3222c] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823c500 [3/c893d42f-80cd-48db-be84-9a6dfee3222c] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823cc80 [3/a51c283d-16da-4e8f-a80b-33be6eb1572b] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823cc80 [3/a51c283d-16da-4e8f-a80b-33be6eb1572b] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823d180 [3/b2460f66-10b1-46c5-80cc-386b73158112] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823d180 [3/b2460f66-10b1-46c5-80cc-386b73158112] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823c780 [3/899fe326-3145-4110-8d10-4de3c934bda0] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.670+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823c780 [3/899fe326-3145-4110-8d10-4de3c934bda0] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.671+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823c280 [3/4f363a97-9a20-4913-a174-4d1fab8a425e] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.671+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823c280 [3/4f363a97-9a20-4913-a174-4d1fab8a425e] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.671+0000 7eff89fe2640 -1 rbd::mirror::ImageReplayer: 0x56401823d400 [3/f0fc7c92-9985-462d-8426-1d934c12daa6] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:18:13.671+0000 7eff8a7e3640 -1 rbd::mirror::ImageReplayer: 0x56401823d400 [3/f0fc7c92-9985-462d-8426-1d934c12daa6] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.678+0000 7ff497c2b640 -1 rbd::mirror::ImageReplayer: 0x55c392259180 [3/bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.678+0000 7ff49842c640 -1 rbd::mirror::ImageReplayer: 0x55c392259180 [3/bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.678+0000 7ff497c2b640 -1 rbd::mirror::ImageReplayer: 0x55c392258c80 [3/af0461fa-e6c8-4e21-90df-d164bb810e1c] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T01:18:13.678+0000 7ff49842c640 -1 rbd::mirror::ImageReplayer: 0x55c392258c80 [3/af0461fa-e6c8-4e21-90df-d164bb810e1c] register_admin_socket_hook: error registering admin socket commands 2024-02-10T01:18:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1714683904 bytes 2024-02-10T01:18:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1734276096 bytes 2024-02-10T01:18:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1759949824 bytes 2024-02-10T01:18:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1828927488 bytes 2024-02-10T01:18:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1906985984 bytes 2024-02-10T01:18:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 1938892800 bytes 2024-02-10T01:18:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 1940484096 bytes 2024-02-10T01:18:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 1988246528 bytes 2024-02-10T01:18:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2040391680 bytes 2024-02-10T01:18:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2068598784 bytes 2024-02-10T01:18:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2086435840 bytes 2024-02-10T01:18:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2124086272 bytes 2024-02-10T01:18:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2196256768 bytes 2024-02-10T01:18:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2261995520 bytes 2024-02-10T01:18:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2297893888 bytes 2024-02-10T01:18:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2406076416 bytes 2024-02-10T01:18:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2485226496 bytes 2024-02-10T01:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2496208896 bytes 2024-02-10T01:18:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2553218048 bytes 2024-02-10T01:18:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2567667712 bytes 2024-02-10T01:18:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2614319104 bytes 2024-02-10T01:18:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2718129152 bytes 2024-02-10T01:18:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2749171712 bytes 2024-02-10T01:18:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2841918464 bytes 2024-02-10T01:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2911994880 bytes 2024-02-10T01:18:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3016519680 bytes 2024-02-10T01:18:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3037169664 bytes 2024-02-10T01:18:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3072030720 bytes 2024-02-10T01:18:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3113655296 bytes 2024-02-10T01:18:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3136182272 bytes 2024-02-10T01:18:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3167312896 bytes 2024-02-10T01:18:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3195815936 bytes 2024-02-10T01:18:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3209945088 bytes 2024-02-10T01:18:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3270585344 bytes 2024-02-10T01:18:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3302148096 bytes 2024-02-10T01:18:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:18:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:18:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_27 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:18:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_27 128M 2024-02-10T01:18:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2024-02-10T01:18:18.509 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-02-10T01:18:18.509 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:18:18.509 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-02-10T01:18:18.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-02-10T01:18:18.510 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-02-10T01:18:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:18:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:18:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-02-10T01:18:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:18:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-02-10T01:18:18.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-02-10T01:18:18.514 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-02-10T01:18:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:18:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_27': 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13b789eba552 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13b789eba552 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:18 2024 2024-02-10T01:18:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:18 2024 2024-02-10T01:18:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:18 2024 2024-02-10T01:18:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:18:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:18:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:18:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:18:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:18:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_27 2024-02-10T01:18:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:18:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:18:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:18:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:18:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:18:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 48301056 bytes 2024-02-10T01:18:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 78596096 bytes 2024-02-10T01:18:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 132722688 bytes 2024-02-10T01:18:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 161120256 bytes 2024-02-10T01:18:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 207938560 bytes 2024-02-10T01:18:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 222230528 bytes 2024-02-10T01:18:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 271448064 bytes 2024-02-10T01:18:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 307655680 bytes 2024-02-10T01:18:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 362370048 bytes 2024-02-10T01:18:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 407927808 bytes 2024-02-10T01:18:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 434362368 bytes 2024-02-10T01:18:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 454059008 bytes 2024-02-10T01:18:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 519540736 bytes 2024-02-10T01:18:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 560508928 bytes 2024-02-10T01:18:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 604326912 bytes 2024-02-10T01:18:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 679759872 bytes 2024-02-10T01:18:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 680349696 bytes 2024-02-10T01:18:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 694718464 bytes 2024-02-10T01:18:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 731198464 bytes 2024-02-10T01:18:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 776171520 bytes 2024-02-10T01:18:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 830362624 bytes 2024-02-10T01:18:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 898932736 bytes 2024-02-10T01:18:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 939867136 bytes 2024-02-10T01:18:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 949889024 bytes 2024-02-10T01:18:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 993982464 bytes 2024-02-10T01:18:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1086536704 bytes 2024-02-10T01:18:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1105826816 bytes 2024-02-10T01:18:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1145858048 bytes 2024-02-10T01:18:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1224632320 bytes 2024-02-10T01:18:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1271040000 bytes 2024-02-10T01:18:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1310412800 bytes 2024-02-10T01:18:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1355602944 bytes 2024-02-10T01:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1423692800 bytes 2024-02-10T01:18:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1437471744 bytes 2024-02-10T01:18:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1497084928 bytes 2024-02-10T01:18:24.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1558277120 bytes 2024-02-10T01:18:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1603206144 bytes 2024-02-10T01:18:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1655486464 bytes 2024-02-10T01:18:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1669833728 bytes 2024-02-10T01:18:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1772962816 bytes 2024-02-10T01:18:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1801605120 bytes 2024-02-10T01:18:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1802230784 bytes 2024-02-10T01:18:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1847536640 bytes 2024-02-10T01:18:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1855415296 bytes 2024-02-10T01:18:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1872228352 bytes 2024-02-10T01:18:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1939208192 bytes 2024-02-10T01:18:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1941271552 bytes 2024-02-10T01:18:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1968233472 bytes 2024-02-10T01:18:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2012615680 bytes 2024-02-10T01:18:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2013242368 bytes 2024-02-10T01:18:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2062345216 bytes 2024-02-10T01:18:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2130887680 bytes 2024-02-10T01:18:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2200913920 bytes 2024-02-10T01:18:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2260446208 bytes 2024-02-10T01:18:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2301406208 bytes 2024-02-10T01:18:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2306589696 bytes 2024-02-10T01:18:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2361433088 bytes 2024-02-10T01:18:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2390834176 bytes 2024-02-10T01:18:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2396257280 bytes 2024-02-10T01:18:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2432270336 bytes 2024-02-10T01:18:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2493496320 bytes 2024-02-10T01:18:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2603537408 bytes 2024-02-10T01:18:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2624150528 bytes 2024-02-10T01:18:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2658731008 bytes 2024-02-10T01:18:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2698068992 bytes 2024-02-10T01:18:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2740070400 bytes 2024-02-10T01:18:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2787920896 bytes 2024-02-10T01:18:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2800842752 bytes 2024-02-10T01:18:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2904478720 bytes 2024-02-10T01:18:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3002974208 bytes 2024-02-10T01:18:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3039513600 bytes 2024-02-10T01:18:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3064228864 bytes 2024-02-10T01:18:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3127663616 bytes 2024-02-10T01:18:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3180221440 bytes 2024-02-10T01:18:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3225328640 bytes 2024-02-10T01:18:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3258510336 bytes 2024-02-10T01:18:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3302317056 bytes 2024-02-10T01:18:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3329494016 bytes 2024-02-10T01:18:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3406476288 bytes 2024-02-10T01:18:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3440187392 bytes 2024-02-10T01:18:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3457884160 bytes 2024-02-10T01:18:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3468634112 bytes 2024-02-10T01:18:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3498253312 bytes 2024-02-10T01:18:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3553042432 bytes 2024-02-10T01:18:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3555093504 bytes 2024-02-10T01:18:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3608217600 bytes 2024-02-10T01:18:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3686025216 bytes 2024-02-10T01:18:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3694192640 bytes 2024-02-10T01:18:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3752834048 bytes 2024-02-10T01:18:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3825848320 bytes 2024-02-10T01:18:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3879277568 bytes 2024-02-10T01:18:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3945519104 bytes 2024-02-10T01:18:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3960536064 bytes 2024-02-10T01:18:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3961160704 bytes 2024-02-10T01:18:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3966278656 bytes 2024-02-10T01:18:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3984472064 bytes 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_28 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_28 128M 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:18:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:18:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:18:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-02-10T01:18:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:18:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-02-10T01:18:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_28': 2024-02-10T01:18:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:18:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:18:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:18:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13c6f24c6caa 2024-02-10T01:18:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13c6f24c6caa 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:32 2024 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:32 2024 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:32 2024 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:18:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:18:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:18:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_28 2024-02-10T01:18:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:18:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:18:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=12 2024-02-10T01:18:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:18:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:18:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 54261760 bytes 2024-02-10T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 102560768 bytes 2024-02-10T01:18:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 118154240 bytes 2024-02-10T01:18:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 198915072 bytes 2024-02-10T01:18:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 271020032 bytes 2024-02-10T01:18:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 332089344 bytes 2024-02-10T01:18:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 381792256 bytes 2024-02-10T01:18:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 404503552 bytes 2024-02-10T01:18:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 459353088 bytes 2024-02-10T01:18:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 497236992 bytes 2024-02-10T01:18:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 566137856 bytes 2024-02-10T01:18:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 573277184 bytes 2024-02-10T01:18:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 617632768 bytes 2024-02-10T01:18:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 690681856 bytes 2024-02-10T01:18:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 748160000 bytes 2024-02-10T01:18:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 792092672 bytes 2024-02-10T01:18:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 800335872 bytes 2024-02-10T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 864621568 bytes 2024-02-10T01:18:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 895382528 bytes 2024-02-10T01:18:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 917193728 bytes 2024-02-10T01:18:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 969112576 bytes 2024-02-10T01:18:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1018257408 bytes 2024-02-10T01:18:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1037979648 bytes 2024-02-10T01:18:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1089767424 bytes 2024-02-10T01:18:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1125983232 bytes 2024-02-10T01:18:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1141342208 bytes 2024-02-10T01:18:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1170290688 bytes 2024-02-10T01:18:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1205483520 bytes 2024-02-10T01:18:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1257159680 bytes 2024-02-10T01:18:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1286364160 bytes 2024-02-10T01:18:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1311667200 bytes 2024-02-10T01:18:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1375748096 bytes 2024-02-10T01:18:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1429568512 bytes 2024-02-10T01:18:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1441404928 bytes 2024-02-10T01:18:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1537258496 bytes 2024-02-10T01:18:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1570623488 bytes 2024-02-10T01:18:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1596067840 bytes 2024-02-10T01:18:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1666806784 bytes 2024-02-10T01:18:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1701653504 bytes 2024-02-10T01:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1725930496 bytes 2024-02-10T01:18:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1756644352 bytes 2024-02-10T01:18:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1784660992 bytes 2024-02-10T01:18:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1807638528 bytes 2024-02-10T01:18:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1869241344 bytes 2024-02-10T01:18:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1912364032 bytes 2024-02-10T01:18:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1938694144 bytes 2024-02-10T01:18:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1988901888 bytes 2024-02-10T01:18:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2044569600 bytes 2024-02-10T01:18:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2070863872 bytes 2024-02-10T01:18:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2134684672 bytes 2024-02-10T01:18:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2190972928 bytes 2024-02-10T01:18:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2232451072 bytes 2024-02-10T01:18:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2260733952 bytes 2024-02-10T01:18:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2303062016 bytes 2024-02-10T01:18:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2360318976 bytes 2024-02-10T01:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2390245376 bytes 2024-02-10T01:18:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2427305984 bytes 2024-02-10T01:18:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2492002304 bytes 2024-02-10T01:18:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2527639552 bytes 2024-02-10T01:18:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2601762816 bytes 2024-02-10T01:18:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2650620928 bytes 2024-02-10T01:18:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2686966784 bytes 2024-02-10T01:18:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2689770496 bytes 2024-02-10T01:18:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2690396160 bytes 2024-02-10T01:18:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2694219776 bytes 2024-02-10T01:18:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2718970880 bytes 2024-02-10T01:18:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2728314880 bytes 2024-02-10T01:18:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2728946688 bytes 2024-02-10T01:18:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2767848448 bytes 2024-02-10T01:18:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2796512256 bytes 2024-02-10T01:18:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2820556800 bytes 2024-02-10T01:18:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2894442496 bytes 2024-02-10T01:18:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2921546752 bytes 2024-02-10T01:18:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3024986112 bytes 2024-02-10T01:18:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3048121344 bytes 2024-02-10T01:18:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3074846720 bytes 2024-02-10T01:18:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3097412608 bytes 2024-02-10T01:18:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3159891968 bytes 2024-02-10T01:18:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3256131584 bytes 2024-02-10T01:18:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3272987648 bytes 2024-02-10T01:18:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3327237120 bytes 2024-02-10T01:18:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3382407168 bytes 2024-02-10T01:18:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3423314944 bytes 2024-02-10T01:18:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3448743936 bytes 2024-02-10T01:18:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3457270784 bytes 2024-02-10T01:18:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3511849984 bytes 2024-02-10T01:18:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3560175616 bytes 2024-02-10T01:18:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3597800448 bytes 2024-02-10T01:18:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_29 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:18:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_29 128M 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:18:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-02-10T01:18:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:18:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-02-10T01:18:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_29': 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13cea9307077 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13cea9307077 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:44 2024 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:44 2024 2024-02-10T01:18:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:44 2024 2024-02-10T01:18:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:18:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:18:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:18:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:18:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:18:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_29 2024-02-10T01:18:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:18:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:18:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=14 2024-02-10T01:18:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:18:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:18:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 36131840 bytes 2024-02-10T01:18:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 55205888 bytes 2024-02-10T01:18:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 110871552 bytes 2024-02-10T01:18:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 111469568 bytes 2024-02-10T01:18:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 139739136 bytes 2024-02-10T01:18:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 201896960 bytes 2024-02-10T01:18:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 252493824 bytes 2024-02-10T01:18:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 309406720 bytes 2024-02-10T01:18:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 337852416 bytes 2024-02-10T01:18:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 379426816 bytes 2024-02-10T01:18:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 456418304 bytes 2024-02-10T01:18:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 457048064 bytes 2024-02-10T01:18:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 493528064 bytes 2024-02-10T01:18:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 571997184 bytes 2024-02-10T01:18:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 580390912 bytes 2024-02-10T01:18:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 604127232 bytes 2024-02-10T01:18:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 614873088 bytes 2024-02-10T01:18:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 674202624 bytes 2024-02-10T01:18:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 719198208 bytes 2024-02-10T01:18:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 748207104 bytes 2024-02-10T01:18:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 798278656 bytes 2024-02-10T01:18:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 835156992 bytes 2024-02-10T01:18:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 885226496 bytes 2024-02-10T01:18:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 937512960 bytes 2024-02-10T01:18:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 958624768 bytes 2024-02-10T01:18:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1025975296 bytes 2024-02-10T01:18:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1054821376 bytes 2024-02-10T01:18:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1058826240 bytes 2024-02-10T01:18:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1124963328 bytes 2024-02-10T01:18:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1157995520 bytes 2024-02-10T01:18:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1228286976 bytes 2024-02-10T01:18:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1273532416 bytes 2024-02-10T01:18:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1298530304 bytes 2024-02-10T01:18:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1328260096 bytes 2024-02-10T01:18:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1342966784 bytes 2024-02-10T01:18:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1398942720 bytes 2024-02-10T01:18:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1481439232 bytes 2024-02-10T01:18:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1533105152 bytes 2024-02-10T01:18:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1579581440 bytes 2024-02-10T01:18:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1609009152 bytes 2024-02-10T01:18:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1646612480 bytes 2024-02-10T01:18:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1677537280 bytes 2024-02-10T01:18:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1723031552 bytes 2024-02-10T01:18:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1772736512 bytes 2024-02-10T01:18:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1837199360 bytes 2024-02-10T01:18:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1921865728 bytes 2024-02-10T01:18:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1954320384 bytes 2024-02-10T01:18:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1980935168 bytes 2024-02-10T01:18:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1988244480 bytes 2024-02-10T01:18:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2044924928 bytes 2024-02-10T01:18:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2071374848 bytes 2024-02-10T01:18:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2118969344 bytes 2024-02-10T01:18:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2167114752 bytes 2024-02-10T01:18:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2167717888 bytes 2024-02-10T01:18:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2211095552 bytes 2024-02-10T01:18:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2237245440 bytes 2024-02-10T01:18:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2307765248 bytes 2024-02-10T01:18:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2377220096 bytes 2024-02-10T01:18:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2393885696 bytes 2024-02-10T01:18:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2451204096 bytes 2024-02-10T01:18:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2505262080 bytes 2024-02-10T01:18:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2546031616 bytes 2024-02-10T01:18:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2649204736 bytes 2024-02-10T01:18:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2746208256 bytes 2024-02-10T01:18:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2778865664 bytes 2024-02-10T01:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2810849280 bytes 2024-02-10T01:18:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2826307584 bytes 2024-02-10T01:18:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2868584448 bytes 2024-02-10T01:18:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2870065152 bytes 2024-02-10T01:18:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2937188352 bytes 2024-02-10T01:18:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2937809920 bytes 2024-02-10T01:18:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2975499264 bytes 2024-02-10T01:18:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3032186880 bytes 2024-02-10T01:18:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3050373120 bytes 2024-02-10T01:18:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3152032768 bytes 2024-02-10T01:18:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3165121536 bytes 2024-02-10T01:18:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3247152128 bytes 2024-02-10T01:18:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3278801920 bytes 2024-02-10T01:18:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3294351360 bytes 2024-02-10T01:18:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3320048640 bytes 2024-02-10T01:18:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3347125248 bytes 2024-02-10T01:18:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3362556928 bytes 2024-02-10T01:18:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3411040256 bytes 2024-02-10T01:18:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3453993984 bytes 2024-02-10T01:18:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3496928256 bytes 2024-02-10T01:18:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3540379648 bytes 2024-02-10T01:18:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3569096704 bytes 2024-02-10T01:18:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3571003392 bytes 2024-02-10T01:18:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3609954304 bytes 2024-02-10T01:18:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3627911168 bytes 2024-02-10T01:18:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3653132288 bytes 2024-02-10T01:18:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3723259904 bytes 2024-02-10T01:18:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3809170432 bytes 2024-02-10T01:18:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3895633920 bytes 2024-02-10T01:18:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3933901824 bytes 2024-02-10T01:18:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3996099584 bytes 2024-02-10T01:18:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 4002417664 bytes 2024-02-10T01:18:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 4031674368 bytes 2024-02-10T01:18:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 4039915520 bytes 2024-02-10T01:18:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 4055674880 bytes 2024-02-10T01:18:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 4107249664 bytes 2024-02-10T01:18:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 4157729792 bytes 2024-02-10T01:18:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 4208602112 bytes 2024-02-10T01:18:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:18:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:18:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_30 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_30 128M 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:18:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:18:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:18:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-02-10T01:18:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:18:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-02-10T01:18:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_30': 2024-02-10T01:18:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:18:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:18:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:18:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13d6876dca6f 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13d6876dca6f 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:59 2024 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:59 2024 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:59 2024 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:18:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:18:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:18:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_30 2024-02-10T01:18:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:18:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:18:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:18:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:18:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:18:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:18:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:19:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 32982016 bytes 2024-02-10T01:19:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 65199104 bytes 2024-02-10T01:19:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 125898752 bytes 2024-02-10T01:19:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 172949504 bytes 2024-02-10T01:19:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 271546368 bytes 2024-02-10T01:19:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 287471616 bytes 2024-02-10T01:19:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 396497920 bytes 2024-02-10T01:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 495195136 bytes 2024-02-10T01:19:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 586870784 bytes 2024-02-10T01:19:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 656681984 bytes 2024-02-10T01:19:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 714158080 bytes 2024-02-10T01:19:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 714752000 bytes 2024-02-10T01:19:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 763454464 bytes 2024-02-10T01:19:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 804808704 bytes 2024-02-10T01:19:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 829576192 bytes 2024-02-10T01:19:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 953799680 bytes 2024-02-10T01:19:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 1008676864 bytes 2024-02-10T01:19:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 1071481856 bytes 2024-02-10T01:19:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 1084808192 bytes 2024-02-10T01:19:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1166577664 bytes 2024-02-10T01:19:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1175488512 bytes 2024-02-10T01:19:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1237039104 bytes 2024-02-10T01:19:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1361739776 bytes 2024-02-10T01:19:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1423629312 bytes 2024-02-10T01:19:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1440289792 bytes 2024-02-10T01:19:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1453894656 bytes 2024-02-10T01:19:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1543170048 bytes 2024-02-10T01:19:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1607334912 bytes 2024-02-10T01:19:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1678357504 bytes 2024-02-10T01:19:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1697754112 bytes 2024-02-10T01:19:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1746332672 bytes 2024-02-10T01:19:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1836608512 bytes 2024-02-10T01:19:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1860159488 bytes 2024-02-10T01:19:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1907167232 bytes 2024-02-10T01:19:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1927000064 bytes 2024-02-10T01:19:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1947270144 bytes 2024-02-10T01:19:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 2028063744 bytes 2024-02-10T01:19:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 2063863808 bytes 2024-02-10T01:19:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 2088763392 bytes 2024-02-10T01:19:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 2131283968 bytes 2024-02-10T01:19:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 2162172928 bytes 2024-02-10T01:19:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 2196432896 bytes 2024-02-10T01:19:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2241446912 bytes 2024-02-10T01:19:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2277254144 bytes 2024-02-10T01:19:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2327472128 bytes 2024-02-10T01:19:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2328079360 bytes 2024-02-10T01:19:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2382547968 bytes 2024-02-10T01:19:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2436127744 bytes 2024-02-10T01:19:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2460884992 bytes 2024-02-10T01:19:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2541415424 bytes 2024-02-10T01:19:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2542023680 bytes 2024-02-10T01:19:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2542620672 bytes 2024-02-10T01:19:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2570368000 bytes 2024-02-10T01:19:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2628810752 bytes 2024-02-10T01:19:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2688530432 bytes 2024-02-10T01:19:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2700691456 bytes 2024-02-10T01:19:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2733021184 bytes 2024-02-10T01:19:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2766808064 bytes 2024-02-10T01:19:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2819436544 bytes 2024-02-10T01:19:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2865920000 bytes 2024-02-10T01:19:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2892939264 bytes 2024-02-10T01:19:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2950444032 bytes 2024-02-10T01:19:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2967392256 bytes 2024-02-10T01:19:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 3025811456 bytes 2024-02-10T01:19:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 3044532224 bytes 2024-02-10T01:19:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3107818496 bytes 2024-02-10T01:19:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3174349824 bytes 2024-02-10T01:19:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3252943872 bytes 2024-02-10T01:19:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3300029440 bytes 2024-02-10T01:19:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3344664576 bytes 2024-02-10T01:19:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3397881856 bytes 2024-02-10T01:19:10.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.522120753879851) > weight (0.25) 2024-02-10T01:19:10.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7032832385334039) > weight (0.25) 2024-02-10T01:19:10.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7317004637068203) > weight (0.25) 2024-02-10T01:19:10.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9625758425555868) > weight (0.25) 2024-02-10T01:19:10.504 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-02-10T01:19:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3416475648 bytes 2024-02-10T01:19:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3453184000 bytes 2024-02-10T01:19:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3463547904 bytes 2024-02-10T01:19:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3468751872 bytes 2024-02-10T01:19:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_31 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_31 128M 2024-02-10T01:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:19:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-02-10T01:19:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:19:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-02-10T01:19:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_31': 2024-02-10T01:19:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13dfeab14aa7 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13dfeab14aa7 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:10 2024 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:10 2024 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:10 2024 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:19:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:19:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:19:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_31 2024-02-10T01:19:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:19:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:19:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:19:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:19:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:19:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 57475072 bytes 2024-02-10T01:19:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 67139584 bytes 2024-02-10T01:19:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 110406656 bytes 2024-02-10T01:19:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 126315520 bytes 2024-02-10T01:19:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 152008704 bytes 2024-02-10T01:19:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 212422656 bytes 2024-02-10T01:19:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 285947904 bytes 2024-02-10T01:19:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 308028416 bytes 2024-02-10T01:19:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 308631552 bytes 2024-02-10T01:19:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 333787136 bytes 2024-02-10T01:19:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 335178752 bytes 2024-02-10T01:19:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 418719744 bytes 2024-02-10T01:19:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 482214912 bytes 2024-02-10T01:19:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 536160256 bytes 2024-02-10T01:19:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 581493760 bytes 2024-02-10T01:19:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 596753408 bytes 2024-02-10T01:19:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 646765568 bytes 2024-02-10T01:19:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 670350336 bytes 2024-02-10T01:19:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 689327104 bytes 2024-02-10T01:19:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 714195968 bytes 2024-02-10T01:19:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 714821632 bytes 2024-02-10T01:19:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 740721664 bytes 2024-02-10T01:19:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 821395456 bytes 2024-02-10T01:19:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 839030784 bytes 2024-02-10T01:19:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 900570112 bytes 2024-02-10T01:19:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 923074560 bytes 2024-02-10T01:19:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 985082880 bytes 2024-02-10T01:19:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1043992576 bytes 2024-02-10T01:19:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1053833216 bytes 2024-02-10T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1095497728 bytes 2024-02-10T01:19:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1134085120 bytes 2024-02-10T01:19:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1171483648 bytes 2024-02-10T01:19:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1193653248 bytes 2024-02-10T01:19:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1231257600 bytes 2024-02-10T01:19:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1248935936 bytes 2024-02-10T01:19:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1297003520 bytes 2024-02-10T01:19:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1318758400 bytes 2024-02-10T01:19:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1343517696 bytes 2024-02-10T01:19:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1369400320 bytes 2024-02-10T01:19:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1392884736 bytes 2024-02-10T01:19:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1479317504 bytes 2024-02-10T01:19:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1516467200 bytes 2024-02-10T01:19:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1517089792 bytes 2024-02-10T01:19:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1577684992 bytes 2024-02-10T01:19:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1657916416 bytes 2024-02-10T01:19:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1699590144 bytes 2024-02-10T01:19:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1710545920 bytes 2024-02-10T01:19:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1776681984 bytes 2024-02-10T01:19:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1838227456 bytes 2024-02-10T01:19:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1895497728 bytes 2024-02-10T01:19:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1925723136 bytes 2024-02-10T01:19:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1960499200 bytes 2024-02-10T01:19:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 1976066048 bytes 2024-02-10T01:19:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 1983356928 bytes 2024-02-10T01:19:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2017394688 bytes 2024-02-10T01:19:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2037217280 bytes 2024-02-10T01:19:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2068259840 bytes 2024-02-10T01:19:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2090359808 bytes 2024-02-10T01:19:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2108947456 bytes 2024-02-10T01:19:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2176686080 bytes 2024-02-10T01:19:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2217571328 bytes 2024-02-10T01:19:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2242395136 bytes 2024-02-10T01:19:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2246324224 bytes 2024-02-10T01:19:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2269908992 bytes 2024-02-10T01:19:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2306032640 bytes 2024-02-10T01:19:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2315871232 bytes 2024-02-10T01:19:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2380165120 bytes 2024-02-10T01:19:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2404005888 bytes 2024-02-10T01:19:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2490985472 bytes 2024-02-10T01:19:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2491614208 bytes 2024-02-10T01:19:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2570591232 bytes 2024-02-10T01:19:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2651967488 bytes 2024-02-10T01:19:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2692061184 bytes 2024-02-10T01:19:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 2768041984 bytes 2024-02-10T01:19:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 2812692480 bytes 2024-02-10T01:19:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 2827929600 bytes 2024-02-10T01:19:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 2854536192 bytes 2024-02-10T01:19:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 2864499712 bytes 2024-02-10T01:19:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 2910135296 bytes 2024-02-10T01:19:22.225 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:19:22.224+0000 7febc0775640 -1 reset not still connected to 0x561ec4716820 2024-02-10T01:19:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 2917029888 bytes 2024-02-10T01:19:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 2950675456 bytes 2024-02-10T01:19:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 2969604096 bytes 2024-02-10T01:19:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 2982033408 bytes 2024-02-10T01:19:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3065899008 bytes 2024-02-10T01:19:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3095692288 bytes 2024-02-10T01:19:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3119607808 bytes 2024-02-10T01:19:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3198429184 bytes 2024-02-10T01:19:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3248935936 bytes 2024-02-10T01:19:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3290442752 bytes 2024-02-10T01:19:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3355764736 bytes 2024-02-10T01:19:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3370717184 bytes 2024-02-10T01:19:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3432610816 bytes 2024-02-10T01:19:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3468127232 bytes 2024-02-10T01:19:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3549074432 bytes 2024-02-10T01:19:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3580648448 bytes 2024-02-10T01:19:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3697470464 bytes 2024-02-10T01:19:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 3778958336 bytes 2024-02-10T01:19:24.868 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:19:24.867+0000 7febc0775640 -1 reset not still connected to 0x561ec4716820 2024-02-10T01:19:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 3800308736 bytes 2024-02-10T01:19:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_32 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_32 128M 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-02-10T01:19:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:19:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_32': 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13e8cbd32c45 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13e8cbd32c45 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:24 2024 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:24 2024 2024-02-10T01:19:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:24 2024 2024-02-10T01:19:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:19:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:19:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:19:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:19:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:19:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_32 2024-02-10T01:19:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:19:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:19:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:19:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:19:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:19:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 31787008 bytes 2024-02-10T01:19:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 102815744 bytes 2024-02-10T01:19:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 142687232 bytes 2024-02-10T01:19:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 207113216 bytes 2024-02-10T01:19:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 298470400 bytes 2024-02-10T01:19:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 335277056 bytes 2024-02-10T01:19:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 335899648 bytes 2024-02-10T01:19:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 363187200 bytes 2024-02-10T01:19:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 387272704 bytes 2024-02-10T01:19:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 455690240 bytes 2024-02-10T01:19:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 491774976 bytes 2024-02-10T01:19:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 526714880 bytes 2024-02-10T01:19:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 567576576 bytes 2024-02-10T01:19:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 579868672 bytes 2024-02-10T01:19:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 642121728 bytes 2024-02-10T01:19:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 657258496 bytes 2024-02-10T01:19:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 680372224 bytes 2024-02-10T01:19:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 704617472 bytes 2024-02-10T01:19:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 792728576 bytes 2024-02-10T01:19:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 831668224 bytes 2024-02-10T01:19:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 845497344 bytes 2024-02-10T01:19:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 902245376 bytes 2024-02-10T01:19:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 985178112 bytes 2024-02-10T01:19:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1045613568 bytes 2024-02-10T01:19:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1089640448 bytes 2024-02-10T01:19:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1099536384 bytes 2024-02-10T01:19:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1122582528 bytes 2024-02-10T01:19:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1162185728 bytes 2024-02-10T01:19:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1193573376 bytes 2024-02-10T01:19:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1229438976 bytes 2024-02-10T01:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1274028032 bytes 2024-02-10T01:19:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1339878400 bytes 2024-02-10T01:19:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1434558464 bytes 2024-02-10T01:19:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1451691008 bytes 2024-02-10T01:19:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1466338304 bytes 2024-02-10T01:19:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1518700544 bytes 2024-02-10T01:19:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1569442816 bytes 2024-02-10T01:19:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1623390208 bytes 2024-02-10T01:19:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1703366656 bytes 2024-02-10T01:19:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1805794304 bytes 2024-02-10T01:19:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1806452736 bytes 2024-02-10T01:19:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1827907584 bytes 2024-02-10T01:19:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1889317888 bytes 2024-02-10T01:19:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1940254720 bytes 2024-02-10T01:19:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2036445184 bytes 2024-02-10T01:19:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2102525952 bytes 2024-02-10T01:19:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2134588416 bytes 2024-02-10T01:19:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2179013632 bytes 2024-02-10T01:19:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2192412672 bytes 2024-02-10T01:19:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2214129664 bytes 2024-02-10T01:19:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:19:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:19:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:19:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_33 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_33 128M 2024-02-10T01:19:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:19:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:19:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:19:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:19:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:19:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-02-10T01:19:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:19:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-02-10T01:19:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_33': 2024-02-10T01:19:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:19:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:19:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:19:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 13f48e01df22 2024-02-10T01:19:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.13f48e01df22 2024-02-10T01:19:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:19:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:19:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:19:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:19:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:32 2024 2024-02-10T01:19:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:32 2024 2024-02-10T01:19:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:32 2024 2024-02-10T01:19:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:19:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:19:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:19:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:19:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:19:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_33 2024-02-10T01:19:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:19:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:19:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=14 2024-02-10T01:19:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:19:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:19:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 9506816 bytes 2024-02-10T01:19:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 29651968 bytes 2024-02-10T01:19:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 53886976 bytes 2024-02-10T01:19:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 54487040 bytes 2024-02-10T01:19:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 95345664 bytes 2024-02-10T01:19:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 156749824 bytes 2024-02-10T01:19:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 190147584 bytes 2024-02-10T01:19:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 212823040 bytes 2024-02-10T01:19:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 273239040 bytes 2024-02-10T01:19:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 302351360 bytes 2024-02-10T01:19:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 399779840 bytes 2024-02-10T01:19:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 424485888 bytes 2024-02-10T01:19:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 529291264 bytes 2024-02-10T01:19:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 592034816 bytes 2024-02-10T01:19:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 613487616 bytes 2024-02-10T01:19:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 642767872 bytes 2024-02-10T01:19:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 680337408 bytes 2024-02-10T01:19:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 705838080 bytes 2024-02-10T01:19:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 728071168 bytes 2024-02-10T01:19:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 761870336 bytes 2024-02-10T01:19:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 815737856 bytes 2024-02-10T01:19:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 864291840 bytes 2024-02-10T01:19:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 872771584 bytes 2024-02-10T01:19:36.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 931153920 bytes 2024-02-10T01:19:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 969421824 bytes 2024-02-10T01:19:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1003187200 bytes 2024-02-10T01:19:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1056369664 bytes 2024-02-10T01:19:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1088534528 bytes 2024-02-10T01:19:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1139768320 bytes 2024-02-10T01:19:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1195281408 bytes 2024-02-10T01:19:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1214077952 bytes 2024-02-10T01:19:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1247178752 bytes 2024-02-10T01:19:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1268883456 bytes 2024-02-10T01:19:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1305773056 bytes 2024-02-10T01:19:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1365357568 bytes 2024-02-10T01:19:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1438053376 bytes 2024-02-10T01:19:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1478197248 bytes 2024-02-10T01:19:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1525363712 bytes 2024-02-10T01:19:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1576238080 bytes 2024-02-10T01:19:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1598072832 bytes 2024-02-10T01:19:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1621857280 bytes 2024-02-10T01:19:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1662690304 bytes 2024-02-10T01:19:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1722874880 bytes 2024-02-10T01:19:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1738123264 bytes 2024-02-10T01:19:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1769060352 bytes 2024-02-10T01:19:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1781951488 bytes 2024-02-10T01:19:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1805237248 bytes 2024-02-10T01:19:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1836375040 bytes 2024-02-10T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1836993536 bytes 2024-02-10T01:19:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 1868745728 bytes 2024-02-10T01:19:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 1903476736 bytes 2024-02-10T01:19:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 1945401344 bytes 2024-02-10T01:19:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 1967758336 bytes 2024-02-10T01:19:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 1997967360 bytes 2024-02-10T01:19:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2040343552 bytes 2024-02-10T01:19:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2040953856 bytes 2024-02-10T01:19:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2118464512 bytes 2024-02-10T01:19:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2183923712 bytes 2024-02-10T01:19:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2201586688 bytes 2024-02-10T01:19:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2216373248 bytes 2024-02-10T01:19:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2219945984 bytes 2024-02-10T01:19:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2253249536 bytes 2024-02-10T01:19:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2303156224 bytes 2024-02-10T01:19:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2369977344 bytes 2024-02-10T01:19:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2423324672 bytes 2024-02-10T01:19:41.514 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:19:41.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:19:41.515 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-02-10T01:19:41.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:19:41.515+0000 7eff8afe4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 43653) UID: 1000 2024-02-10T01:19:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2463655936 bytes 2024-02-10T01:19:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2525313024 bytes 2024-02-10T01:19:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2600798208 bytes 2024-02-10T01:19:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2645898240 bytes 2024-02-10T01:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2688289792 bytes 2024-02-10T01:19:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2704660480 bytes 2024-02-10T01:19:42.516 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:19:42.516 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:19:42.516 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:19:42.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:19:42.517 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:19:42.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:19:42.520 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-02-10T01:19:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2729139200 bytes 2024-02-10T01:19:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2755849216 bytes 2024-02-10T01:19:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 2805816320 bytes 2024-02-10T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 2830141440 bytes 2024-02-10T01:19:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 2867555328 bytes 2024-02-10T01:19:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 2925041664 bytes 2024-02-10T01:19:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 2980404224 bytes 2024-02-10T01:19:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3029449728 bytes 2024-02-10T01:19:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3078470656 bytes 2024-02-10T01:19:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3128541184 bytes 2024-02-10T01:19:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3149538304 bytes 2024-02-10T01:19:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3194645504 bytes 2024-02-10T01:19:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3224925184 bytes 2024-02-10T01:19:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3283857408 bytes 2024-02-10T01:19:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3340792832 bytes 2024-02-10T01:19:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3388451840 bytes 2024-02-10T01:19:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3410587648 bytes 2024-02-10T01:19:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3454861312 bytes 2024-02-10T01:19:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 3455472640 bytes 2024-02-10T01:19:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 3472485376 bytes 2024-02-10T01:19:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 3518544896 bytes 2024-02-10T01:19:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 3519821824 bytes 2024-02-10T01:19:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9400 IOs, 3608500224 bytes 2024-02-10T01:19:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9500 IOs, 3647711232 bytes 2024-02-10T01:19:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9600 IOs, 3704153088 bytes 2024-02-10T01:19:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9700 IOs, 3731671040 bytes 2024-02-10T01:19:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9800 IOs, 3732289536 bytes 2024-02-10T01:19:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9900 IOs, 3779539968 bytes 2024-02-10T01:19:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10000 IOs, 3862628352 bytes 2024-02-10T01:19:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10100 IOs, 3933802496 bytes 2024-02-10T01:19:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10200 IOs, 3983538176 bytes 2024-02-10T01:19:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10300 IOs, 3999526912 bytes 2024-02-10T01:19:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:10400 IOs, 4034273280 bytes 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_34 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_34 128M 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2024-02-10T01:19:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:19:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:19:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-02-10T01:19:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:19:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-02-10T01:19:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:19:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_34': 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1406efaed4eb 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1406efaed4eb 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:46 2024 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:46 2024 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:46 2024 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:19:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:19:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:19:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_34 2024-02-10T01:19:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:19:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:19:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=10 2024-02-10T01:19:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:19:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:19:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 51921920 bytes 2024-02-10T01:19:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 135000064 bytes 2024-02-10T01:19:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 174959616 bytes 2024-02-10T01:19:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 204798976 bytes 2024-02-10T01:19:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 274455552 bytes 2024-02-10T01:19:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 317402112 bytes 2024-02-10T01:19:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 375820288 bytes 2024-02-10T01:19:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 396294144 bytes 2024-02-10T01:19:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 477843456 bytes 2024-02-10T01:19:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 539397120 bytes 2024-02-10T01:19:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 558202880 bytes 2024-02-10T01:19:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 640128000 bytes 2024-02-10T01:19:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 723077120 bytes 2024-02-10T01:19:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 805581824 bytes 2024-02-10T01:19:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 808721408 bytes 2024-02-10T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 809341952 bytes 2024-02-10T01:19:50.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 881698816 bytes 2024-02-10T01:19:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 956730368 bytes 2024-02-10T01:19:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 991104000 bytes 2024-02-10T01:19:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1026550784 bytes 2024-02-10T01:19:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1109367808 bytes 2024-02-10T01:19:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1197221888 bytes 2024-02-10T01:19:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1204850688 bytes 2024-02-10T01:19:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1218112512 bytes 2024-02-10T01:19:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1265062912 bytes 2024-02-10T01:19:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1276585984 bytes 2024-02-10T01:19:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1279886336 bytes 2024-02-10T01:19:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1372448768 bytes 2024-02-10T01:19:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1433950208 bytes 2024-02-10T01:19:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1500958720 bytes 2024-02-10T01:19:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1539431424 bytes 2024-02-10T01:19:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1581803520 bytes 2024-02-10T01:19:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1614936064 bytes 2024-02-10T01:19:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1632330752 bytes 2024-02-10T01:19:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1667395584 bytes 2024-02-10T01:19:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1712889856 bytes 2024-02-10T01:19:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1734746112 bytes 2024-02-10T01:19:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1757755392 bytes 2024-02-10T01:19:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1792290816 bytes 2024-02-10T01:19:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1822651392 bytes 2024-02-10T01:19:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1885158400 bytes 2024-02-10T01:19:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1937757184 bytes 2024-02-10T01:19:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2007322624 bytes 2024-02-10T01:19:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2030644224 bytes 2024-02-10T01:19:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2077446144 bytes 2024-02-10T01:19:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2104549376 bytes 2024-02-10T01:19:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2160272384 bytes 2024-02-10T01:19:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2214331392 bytes 2024-02-10T01:19:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2218031104 bytes 2024-02-10T01:19:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2261539840 bytes 2024-02-10T01:19:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2287229952 bytes 2024-02-10T01:19:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2301883392 bytes 2024-02-10T01:19:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2363534336 bytes 2024-02-10T01:19:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2406508544 bytes 2024-02-10T01:19:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2416798720 bytes 2024-02-10T01:19:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2471851008 bytes 2024-02-10T01:19:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2488256512 bytes 2024-02-10T01:19:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2491721728 bytes 2024-02-10T01:19:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2563892224 bytes 2024-02-10T01:19:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2632203264 bytes 2024-02-10T01:19:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2635041792 bytes 2024-02-10T01:19:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2642970624 bytes 2024-02-10T01:19:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2643597312 bytes 2024-02-10T01:19:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2737955840 bytes 2024-02-10T01:19:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2784295936 bytes 2024-02-10T01:19:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2821944320 bytes 2024-02-10T01:19:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2845264896 bytes 2024-02-10T01:19:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2898046976 bytes 2024-02-10T01:19:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2922632192 bytes 2024-02-10T01:19:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2974854144 bytes 2024-02-10T01:19:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2990843904 bytes 2024-02-10T01:19:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3078670336 bytes 2024-02-10T01:19:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_35 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_35 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:19:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_35 128M 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_35 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:19:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2024-02-10T01:19:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_35 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:19:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-02-10T01:19:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:19:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-02-10T01:19:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_35': 2024-02-10T01:19:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:19:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:19:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:19:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1412777d196e 2024-02-10T01:19:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1412777d196e 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:57 2024 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:57 2024 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:57 2024 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 67ccb183-b9d9-46d5-8a34-38d09683d20e 2024-02-10T01:19:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:19:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:19:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_35 2024-02-10T01:19:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:19:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:19:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_35 2024-02-10T01:19:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:19:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=13 2024-02-10T01:19:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:19:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:19:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 29412352 bytes 2024-02-10T01:19:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 88199168 bytes 2024-02-10T01:19:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 104541184 bytes 2024-02-10T01:19:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 174075904 bytes 2024-02-10T01:19:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 204834816 bytes 2024-02-10T01:19:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 276346880 bytes 2024-02-10T01:20:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 323831808 bytes 2024-02-10T01:20:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 355606528 bytes 2024-02-10T01:20:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 392596480 bytes 2024-02-10T01:20:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 399408128 bytes 2024-02-10T01:20:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 454579200 bytes 2024-02-10T01:20:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 499877888 bytes 2024-02-10T01:20:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 516020224 bytes 2024-02-10T01:20:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 524014592 bytes 2024-02-10T01:20:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 554419200 bytes 2024-02-10T01:20:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 591581184 bytes 2024-02-10T01:20:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 688204800 bytes 2024-02-10T01:20:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 710479872 bytes 2024-02-10T01:20:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 748669952 bytes 2024-02-10T01:20:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 805642240 bytes 2024-02-10T01:20:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 869507072 bytes 2024-02-10T01:20:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 962787328 bytes 2024-02-10T01:20:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 977916928 bytes 2024-02-10T01:20:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1019882496 bytes 2024-02-10T01:20:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1059178496 bytes 2024-02-10T01:20:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1137915904 bytes 2024-02-10T01:20:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1138536448 bytes 2024-02-10T01:20:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1166832640 bytes 2024-02-10T01:20:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1185104896 bytes 2024-02-10T01:20:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1228169216 bytes 2024-02-10T01:20:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1297801216 bytes 2024-02-10T01:20:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1336054784 bytes 2024-02-10T01:20:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1359519744 bytes 2024-02-10T01:20:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1381524480 bytes 2024-02-10T01:20:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1404163072 bytes 2024-02-10T01:20:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1425526784 bytes 2024-02-10T01:20:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1474015232 bytes 2024-02-10T01:20:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1567670272 bytes 2024-02-10T01:20:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1584940032 bytes 2024-02-10T01:20:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1630434304 bytes 2024-02-10T01:20:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1723038720 bytes 2024-02-10T01:20:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1746735104 bytes 2024-02-10T01:20:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1801574400 bytes 2024-02-10T01:20:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1847189504 bytes 2024-02-10T01:20:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1896529920 bytes 2024-02-10T01:20:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1951413248 bytes 2024-02-10T01:20:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1977072640 bytes 2024-02-10T01:20:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2000352256 bytes 2024-02-10T01:20:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2070040576 bytes 2024-02-10T01:20:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2125492224 bytes 2024-02-10T01:20:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2162091008 bytes 2024-02-10T01:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2238520320 bytes 2024-02-10T01:20:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2301434880 bytes 2024-02-10T01:20:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2408766464 bytes 2024-02-10T01:20:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2430928896 bytes 2024-02-10T01:20:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2456817664 bytes 2024-02-10T01:20:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2483593216 bytes 2024-02-10T01:20:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2537244672 bytes 2024-02-10T01:20:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2637165568 bytes 2024-02-10T01:20:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2686458880 bytes 2024-02-10T01:20:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2755047424 bytes 2024-02-10T01:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2824563712 bytes 2024-02-10T01:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2892717056 bytes 2024-02-10T01:20:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2976781312 bytes 2024-02-10T01:20:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 3097902080 bytes 2024-02-10T01:20:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3133914112 bytes 2024-02-10T01:20:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3153894400 bytes 2024-02-10T01:20:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3181198336 bytes 2024-02-10T01:20:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3253988352 bytes 2024-02-10T01:20:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3282281472 bytes 2024-02-10T01:20:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 3387370496 bytes 2024-02-10T01:20:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3389298688 bytes 2024-02-10T01:20:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3471688704 bytes 2024-02-10T01:20:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3528788992 bytes 2024-02-10T01:20:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3537149952 bytes 2024-02-10T01:20:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3578512384 bytes 2024-02-10T01:20:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3593391104 bytes 2024-02-10T01:20:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3621884928 bytes 2024-02-10T01:20:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3643705344 bytes 2024-02-10T01:20:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3682587648 bytes 2024-02-10T01:20:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3683221504 bytes 2024-02-10T01:20:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3708715008 bytes 2024-02-10T01:20:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3783822336 bytes 2024-02-10T01:20:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8400 IOs, 3807770624 bytes 2024-02-10T01:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8500 IOs, 3857931264 bytes 2024-02-10T01:20:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8600 IOs, 3877300224 bytes 2024-02-10T01:20:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8700 IOs, 3936315392 bytes 2024-02-10T01:20:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8800 IOs, 3978045440 bytes 2024-02-10T01:20:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8900 IOs, 3978644480 bytes 2024-02-10T01:20:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9000 IOs, 4016125952 bytes 2024-02-10T01:20:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9100 IOs, 4047497216 bytes 2024-02-10T01:20:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9200 IOs, 4122611712 bytes 2024-02-10T01:20:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:9300 IOs, 4162129920 bytes 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_36 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_36 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_36 128M 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_36 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:20:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:20:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:20:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2024-02-10T01:20:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:20:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:20:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-02-10T01:20:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_36 2024-02-10T01:20:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-02-10T01:20:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:20:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-02-10T01:20:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_36': 2024-02-10T01:20:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:20:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:20:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:20:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 141eac17e2f 2024-02-10T01:20:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.141eac17e2f 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:12 2024 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:12 2024 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:12 2024 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: ac3ee523-253b-453f-9e13-a4d28ce2c454 2024-02-10T01:20:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_36 2024-02-10T01:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_36 2024-02-10T01:20:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:20:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=10 2024-02-10T01:20:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:20:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:20:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 74324992 bytes 2024-02-10T01:20:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 96753664 bytes 2024-02-10T01:20:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 178767872 bytes 2024-02-10T01:20:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 213003264 bytes 2024-02-10T01:20:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 233530368 bytes 2024-02-10T01:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 255756288 bytes 2024-02-10T01:20:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 291492864 bytes 2024-02-10T01:20:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 339870720 bytes 2024-02-10T01:20:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 354653184 bytes 2024-02-10T01:20:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 424029184 bytes 2024-02-10T01:20:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 459737088 bytes 2024-02-10T01:20:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 509605888 bytes 2024-02-10T01:20:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 556535808 bytes 2024-02-10T01:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 578197504 bytes 2024-02-10T01:20:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 608611328 bytes 2024-02-10T01:20:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 677270528 bytes 2024-02-10T01:20:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 737496064 bytes 2024-02-10T01:20:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 769415168 bytes 2024-02-10T01:20:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 800259072 bytes 2024-02-10T01:20:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 874682368 bytes 2024-02-10T01:20:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 953293824 bytes 2024-02-10T01:20:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 953903104 bytes 2024-02-10T01:20:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1005505536 bytes 2024-02-10T01:20:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1043636224 bytes 2024-02-10T01:20:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1056769024 bytes 2024-02-10T01:20:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1142548480 bytes 2024-02-10T01:20:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1175832576 bytes 2024-02-10T01:20:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1196085248 bytes 2024-02-10T01:20:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1220218880 bytes 2024-02-10T01:20:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1292910592 bytes 2024-02-10T01:20:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1344657408 bytes 2024-02-10T01:20:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1358204928 bytes 2024-02-10T01:20:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1395687424 bytes 2024-02-10T01:20:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1422527488 bytes 2024-02-10T01:20:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1495257088 bytes 2024-02-10T01:20:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1526383616 bytes 2024-02-10T01:20:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1598445568 bytes 2024-02-10T01:20:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1613422592 bytes 2024-02-10T01:20:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1634283520 bytes 2024-02-10T01:20:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1654386688 bytes 2024-02-10T01:20:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1714619392 bytes 2024-02-10T01:20:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1758182400 bytes 2024-02-10T01:20:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1784344576 bytes 2024-02-10T01:20:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1789743104 bytes 2024-02-10T01:20:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1853901824 bytes 2024-02-10T01:20:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1891893248 bytes 2024-02-10T01:20:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1914854400 bytes 2024-02-10T01:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1950470144 bytes 2024-02-10T01:20:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2042602496 bytes 2024-02-10T01:20:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2059911168 bytes 2024-02-10T01:20:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2111413248 bytes 2024-02-10T01:20:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2166562816 bytes 2024-02-10T01:20:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2220593152 bytes 2024-02-10T01:20:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2253692928 bytes 2024-02-10T01:20:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2313515008 bytes 2024-02-10T01:20:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2340377600 bytes 2024-02-10T01:20:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2356422656 bytes 2024-02-10T01:20:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2357037056 bytes 2024-02-10T01:20:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2399457280 bytes 2024-02-10T01:20:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2483125248 bytes 2024-02-10T01:20:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2540682240 bytes 2024-02-10T01:20:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2553828352 bytes 2024-02-10T01:20:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2588611584 bytes 2024-02-10T01:20:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2646526976 bytes 2024-02-10T01:20:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2680397824 bytes 2024-02-10T01:20:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2680996864 bytes 2024-02-10T01:20:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2716186624 bytes 2024-02-10T01:20:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2716791808 bytes 2024-02-10T01:20:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2745027584 bytes 2024-02-10T01:20:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2818701312 bytes 2024-02-10T01:20:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2870144000 bytes 2024-02-10T01:20:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 2884201472 bytes 2024-02-10T01:20:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 2949023744 bytes 2024-02-10T01:20:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 2982400000 bytes 2024-02-10T01:20:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:20:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:20:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:20:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:20:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_37 2024-02-10T01:20:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-02-10T01:20:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_37 2024-02-10T01:20:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_37 128M 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_37 2024-02-10T01:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2024-02-10T01:20:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:20:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:20:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-02-10T01:20:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_37 2024-02-10T01:20:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-02-10T01:20:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:20:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-02-10T01:20:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_37': 2024-02-10T01:20:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:20:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:20:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:20:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 142aa9e9935b 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.142aa9e9935b 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:22 2024 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:22 2024 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:22 2024 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c71c4846-b43a-41df-be57-befe30f9b331 2024-02-10T01:20:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:20:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:20:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_37 2024-02-10T01:20:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_37 2024-02-10T01:20:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:20:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:20:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:20:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:20:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 615424 bytes 2024-02-10T01:20:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 98208768 bytes 2024-02-10T01:20:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 146956288 bytes 2024-02-10T01:20:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 194865152 bytes 2024-02-10T01:20:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 227995648 bytes 2024-02-10T01:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 255183872 bytes 2024-02-10T01:20:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 260907008 bytes 2024-02-10T01:20:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 278840320 bytes 2024-02-10T01:20:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 313046016 bytes 2024-02-10T01:20:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 379745280 bytes 2024-02-10T01:20:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 409790464 bytes 2024-02-10T01:20:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 457574400 bytes 2024-02-10T01:20:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 491954176 bytes 2024-02-10T01:20:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 521976832 bytes 2024-02-10T01:20:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 614241280 bytes 2024-02-10T01:20:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 676004864 bytes 2024-02-10T01:20:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 729533440 bytes 2024-02-10T01:20:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 759170048 bytes 2024-02-10T01:20:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 773558272 bytes 2024-02-10T01:20:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 789678080 bytes 2024-02-10T01:20:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 834870272 bytes 2024-02-10T01:20:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 882338816 bytes 2024-02-10T01:20:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 938612736 bytes 2024-02-10T01:20:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 991010816 bytes 2024-02-10T01:20:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1005264896 bytes 2024-02-10T01:20:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1020588032 bytes 2024-02-10T01:20:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1092770816 bytes 2024-02-10T01:20:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1129573376 bytes 2024-02-10T01:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1182888960 bytes 2024-02-10T01:20:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1189825536 bytes 2024-02-10T01:20:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1249129472 bytes 2024-02-10T01:20:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1293260800 bytes 2024-02-10T01:20:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1322560512 bytes 2024-02-10T01:20:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1367079936 bytes 2024-02-10T01:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1456665600 bytes 2024-02-10T01:20:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1518689280 bytes 2024-02-10T01:20:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1550790656 bytes 2024-02-10T01:20:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1617886208 bytes 2024-02-10T01:20:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1639215104 bytes 2024-02-10T01:20:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1672734720 bytes 2024-02-10T01:20:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1736964096 bytes 2024-02-10T01:20:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1797779456 bytes 2024-02-10T01:20:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1818400768 bytes 2024-02-10T01:20:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1847988224 bytes 2024-02-10T01:20:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1933368320 bytes 2024-02-10T01:20:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2023066624 bytes 2024-02-10T01:20:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2108624896 bytes 2024-02-10T01:20:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2156266496 bytes 2024-02-10T01:20:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2181598208 bytes 2024-02-10T01:20:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2234658816 bytes 2024-02-10T01:20:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2276600832 bytes 2024-02-10T01:20:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2296062976 bytes 2024-02-10T01:20:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2296668160 bytes 2024-02-10T01:20:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2326245376 bytes 2024-02-10T01:20:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2350379008 bytes 2024-02-10T01:20:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2375826432 bytes 2024-02-10T01:20:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2390493184 bytes 2024-02-10T01:20:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2460901376 bytes 2024-02-10T01:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2552305664 bytes 2024-02-10T01:20:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2595322880 bytes 2024-02-10T01:20:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2608922624 bytes 2024-02-10T01:20:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2686098432 bytes 2024-02-10T01:20:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2724173824 bytes 2024-02-10T01:20:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2738849792 bytes 2024-02-10T01:20:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2795717632 bytes 2024-02-10T01:20:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2815514624 bytes 2024-02-10T01:20:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2828481536 bytes 2024-02-10T01:20:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2854956032 bytes 2024-02-10T01:20:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2905863168 bytes 2024-02-10T01:20:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2906460160 bytes 2024-02-10T01:20:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2965945344 bytes 2024-02-10T01:20:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3018614784 bytes 2024-02-10T01:20:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3061451776 bytes 2024-02-10T01:20:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3103023104 bytes 2024-02-10T01:20:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3171396608 bytes 2024-02-10T01:20:34.112 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:20:34.111+0000 7febc0775640 -1 reset not still connected to 0x561ec25cd380 2024-02-10T01:20:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3221733376 bytes 2024-02-10T01:20:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3296669696 bytes 2024-02-10T01:20:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3340277760 bytes 2024-02-10T01:20:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3365203968 bytes 2024-02-10T01:20:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3494122496 bytes 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_38 2024-02-10T01:20:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_38 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_38 128M 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_38 2024-02-10T01:20:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:20:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:20:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:20:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2024-02-10T01:20:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:20:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_38 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-02-10T01:20:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:20:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-02-10T01:20:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_38': 2024-02-10T01:20:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:20:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 143617b9f43d 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.143617b9f43d 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:34 2024 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:34 2024 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:34 2024 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0e8f3781-2b98-408e-b647-688572c90bbb 2024-02-10T01:20:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:20:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:20:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_38 2024-02-10T01:20:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_38 2024-02-10T01:20:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:20:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=5 2024-02-10T01:20:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:20:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:20:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 78445568 bytes 2024-02-10T01:20:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 106873856 bytes 2024-02-10T01:20:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 164577280 bytes 2024-02-10T01:20:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 197984256 bytes 2024-02-10T01:20:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 264367104 bytes 2024-02-10T01:20:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 301282304 bytes 2024-02-10T01:20:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 330484736 bytes 2024-02-10T01:20:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 377409536 bytes 2024-02-10T01:20:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 486755328 bytes 2024-02-10T01:20:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 519320576 bytes 2024-02-10T01:20:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 593140736 bytes 2024-02-10T01:20:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 629945344 bytes 2024-02-10T01:20:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 724515840 bytes 2024-02-10T01:20:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 787914752 bytes 2024-02-10T01:20:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 788524032 bytes 2024-02-10T01:20:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 809564160 bytes 2024-02-10T01:20:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 893554688 bytes 2024-02-10T01:20:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 925810688 bytes 2024-02-10T01:20:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 966208512 bytes 2024-02-10T01:20:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 1023367168 bytes 2024-02-10T01:20:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1051249664 bytes 2024-02-10T01:20:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1093467136 bytes 2024-02-10T01:20:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1133418496 bytes 2024-02-10T01:20:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1134018560 bytes 2024-02-10T01:20:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1150995456 bytes 2024-02-10T01:20:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1157032960 bytes 2024-02-10T01:20:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1267244032 bytes 2024-02-10T01:20:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1303738368 bytes 2024-02-10T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1333063680 bytes 2024-02-10T01:20:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1382940672 bytes 2024-02-10T01:20:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1423840256 bytes 2024-02-10T01:20:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1443072000 bytes 2024-02-10T01:20:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1512428544 bytes 2024-02-10T01:20:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1593427968 bytes 2024-02-10T01:20:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1658261504 bytes 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_39 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_39 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:20:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_39 128M 2024-02-10T01:20:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_39 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2024-02-10T01:20:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:20:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_39 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-02-10T01:20:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:20:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_39': 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1442358d1fe 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1442358d1fe 2024-02-10T01:20:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:40 2024 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:40 2024 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:40 2024 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a70dea2d-caa0-45be-8634-f6297695c64e 2024-02-10T01:20:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:20:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:20:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_39 2024-02-10T01:20:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_39 2024-02-10T01:20:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:20:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:20:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:20:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:20:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 38929408 bytes 2024-02-10T01:20:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 59964416 bytes 2024-02-10T01:20:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 91146240 bytes 2024-02-10T01:20:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 131083264 bytes 2024-02-10T01:20:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 179205120 bytes 2024-02-10T01:20:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 225785856 bytes 2024-02-10T01:20:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 293880832 bytes 2024-02-10T01:20:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 332954624 bytes 2024-02-10T01:20:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 378029056 bytes 2024-02-10T01:20:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 408516608 bytes 2024-02-10T01:20:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 477934592 bytes 2024-02-10T01:20:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 494524416 bytes 2024-02-10T01:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 501253120 bytes 2024-02-10T01:20:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 520480768 bytes 2024-02-10T01:20:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 550987776 bytes 2024-02-10T01:20:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 574137344 bytes 2024-02-10T01:20:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 582223872 bytes 2024-02-10T01:20:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 666235904 bytes 2024-02-10T01:20:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 749457408 bytes 2024-02-10T01:20:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 813270016 bytes 2024-02-10T01:20:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 832844800 bytes 2024-02-10T01:20:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 855448576 bytes 2024-02-10T01:20:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 893478912 bytes 2024-02-10T01:20:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 962844672 bytes 2024-02-10T01:20:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1034312704 bytes 2024-02-10T01:20:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1067857920 bytes 2024-02-10T01:20:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1090085888 bytes 2024-02-10T01:20:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1142772736 bytes 2024-02-10T01:20:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1185673216 bytes 2024-02-10T01:20:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1243117568 bytes 2024-02-10T01:20:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1273049088 bytes 2024-02-10T01:20:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1305274368 bytes 2024-02-10T01:20:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1347780608 bytes 2024-02-10T01:20:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1366453248 bytes 2024-02-10T01:20:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1397389312 bytes 2024-02-10T01:20:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1480993792 bytes 2024-02-10T01:20:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1519883264 bytes 2024-02-10T01:20:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1543894016 bytes 2024-02-10T01:20:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1565531136 bytes 2024-02-10T01:20:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1566154752 bytes 2024-02-10T01:20:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1594940416 bytes 2024-02-10T01:20:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1612115968 bytes 2024-02-10T01:20:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1651404800 bytes 2024-02-10T01:20:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1688431616 bytes 2024-02-10T01:20:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1734473728 bytes 2024-02-10T01:20:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1796847616 bytes 2024-02-10T01:20:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1903906816 bytes 2024-02-10T01:20:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1959078912 bytes 2024-02-10T01:20:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1979939840 bytes 2024-02-10T01:20:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2032779264 bytes 2024-02-10T01:20:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2044530688 bytes 2024-02-10T01:20:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2061674496 bytes 2024-02-10T01:20:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2085616640 bytes 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_40 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_40 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_40 128M 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_40 2024-02-10T01:20:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:20:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:20:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:20:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_40 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-02-10T01:20:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:20:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_40': 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 144e78e5d3d3 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.144e78e5d3d3 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:20:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:47 2024 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:47 2024 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:47 2024 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: fa535af5-2334-4635-9fbe-5c4d54ca6819 2024-02-10T01:20:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:20:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:20:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_40 2024-02-10T01:20:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_40 2024-02-10T01:20:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:20:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=5 2024-02-10T01:20:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:20:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 108468224 bytes 2024-02-10T01:20:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 171228160 bytes 2024-02-10T01:20:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 171848704 bytes 2024-02-10T01:20:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 199386112 bytes 2024-02-10T01:20:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 244391936 bytes 2024-02-10T01:20:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 275520512 bytes 2024-02-10T01:20:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 311010304 bytes 2024-02-10T01:20:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 364124160 bytes 2024-02-10T01:20:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 385721344 bytes 2024-02-10T01:20:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 414807040 bytes 2024-02-10T01:20:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 469460992 bytes 2024-02-10T01:20:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 470096896 bytes 2024-02-10T01:20:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 502155264 bytes 2024-02-10T01:20:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 543022080 bytes 2024-02-10T01:20:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 609937408 bytes 2024-02-10T01:20:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 628059136 bytes 2024-02-10T01:20:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 644395008 bytes 2024-02-10T01:20:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 685863936 bytes 2024-02-10T01:20:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 781622272 bytes 2024-02-10T01:20:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 822002688 bytes 2024-02-10T01:20:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 869366784 bytes 2024-02-10T01:20:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 937143296 bytes 2024-02-10T01:20:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 970361856 bytes 2024-02-10T01:20:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 972256256 bytes 2024-02-10T01:20:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1066125312 bytes 2024-02-10T01:20:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1066738688 bytes 2024-02-10T01:20:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1088487424 bytes 2024-02-10T01:20:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1133846528 bytes 2024-02-10T01:20:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1164205056 bytes 2024-02-10T01:20:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1227457536 bytes 2024-02-10T01:20:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1266293760 bytes 2024-02-10T01:20:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1303014400 bytes 2024-02-10T01:20:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1320551424 bytes 2024-02-10T01:20:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1358406656 bytes 2024-02-10T01:20:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1383319552 bytes 2024-02-10T01:20:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1475978240 bytes 2024-02-10T01:20:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:20:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:20:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_41 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_41 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_41 128M 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_41 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:20:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2024-02-10T01:20:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:20:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_41 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:20:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-02-10T01:20:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:20:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_41': 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 145a438ff098 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.145a438ff098 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:53 2024 2024-02-10T01:20:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:53 2024 2024-02-10T01:20:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:53 2024 2024-02-10T01:20:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:20:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:20:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e8b28d26-f2b8-40d9-8c1e-01e760d124f8 2024-02-10T01:20:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:20:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:20:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_41 2024-02-10T01:20:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:20:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:20:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_41 2024-02-10T01:20:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:20:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=9 2024-02-10T01:20:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:20:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:20:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 63988736 bytes 2024-02-10T01:20:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 115663872 bytes 2024-02-10T01:20:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 181792768 bytes 2024-02-10T01:20:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 214243328 bytes 2024-02-10T01:20:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 293708800 bytes 2024-02-10T01:20:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 362933248 bytes 2024-02-10T01:20:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 442363904 bytes 2024-02-10T01:20:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 465813504 bytes 2024-02-10T01:20:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 487285760 bytes 2024-02-10T01:20:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 503849984 bytes 2024-02-10T01:20:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 538576896 bytes 2024-02-10T01:20:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 600912896 bytes 2024-02-10T01:20:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 620067840 bytes 2024-02-10T01:20:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 674689024 bytes 2024-02-10T01:20:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 707735552 bytes 2024-02-10T01:20:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 708338688 bytes 2024-02-10T01:20:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 777408512 bytes 2024-02-10T01:20:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 819931136 bytes 2024-02-10T01:20:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 859471872 bytes 2024-02-10T01:20:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 895621120 bytes 2024-02-10T01:20:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 942718976 bytes 2024-02-10T01:20:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 968482816 bytes 2024-02-10T01:20:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1004233728 bytes 2024-02-10T01:20:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1021231104 bytes 2024-02-10T01:20:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1054319616 bytes 2024-02-10T01:20:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1169556480 bytes 2024-02-10T01:20:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1207481344 bytes 2024-02-10T01:20:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1303852032 bytes 2024-02-10T01:20:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1326021632 bytes 2024-02-10T01:20:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1352536064 bytes 2024-02-10T01:20:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1399165952 bytes 2024-02-10T01:20:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1424841728 bytes 2024-02-10T01:20:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1450133504 bytes 2024-02-10T01:20:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1461348352 bytes 2024-02-10T01:20:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1485087744 bytes 2024-02-10T01:20:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1496379392 bytes 2024-02-10T01:20:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1569089536 bytes 2024-02-10T01:20:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1609696256 bytes 2024-02-10T01:20:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1657573376 bytes 2024-02-10T01:20:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1684641792 bytes 2024-02-10T01:21:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1758572544 bytes 2024-02-10T01:21:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1800505344 bytes 2024-02-10T01:21:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1819748352 bytes 2024-02-10T01:21:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1834109952 bytes 2024-02-10T01:21:00.520 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:21:00.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:21:00.521 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-02-10T01:21:00.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:21:00.521+0000 7f7b3ac93640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 50316) UID: 1000 2024-02-10T01:21:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1872418816 bytes 2024-02-10T01:21:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1943303168 bytes 2024-02-10T01:21:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1986372608 bytes 2024-02-10T01:21:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2034870272 bytes 2024-02-10T01:21:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2097002496 bytes 2024-02-10T01:21:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2142343168 bytes 2024-02-10T01:21:01.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.34197399528318073) > weight (0.25) 2024-02-10T01:21:01.505 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-02-10T01:21:01.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-02-10T01:21:01.506 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-02-10T01:21:01.506 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-02-10T01:21:01.506 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:21:01.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi060.stderr:2024-02-10T01:21:01.504+0000 7f0c221cb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41602) UID: 1000 2024-02-10T01:21:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2253861888 bytes 2024-02-10T01:21:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2336606208 bytes 2024-02-10T01:21:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2351017984 bytes 2024-02-10T01:21:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2378175488 bytes 2024-02-10T01:21:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2412749824 bytes 2024-02-10T01:21:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2443311104 bytes 2024-02-10T01:21:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2454153216 bytes 2024-02-10T01:21:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2492370944 bytes 2024-02-10T01:21:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2506822656 bytes 2024-02-10T01:21:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2551128064 bytes 2024-02-10T01:21:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2622385152 bytes 2024-02-10T01:21:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2673906688 bytes 2024-02-10T01:21:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2710962176 bytes 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_42 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_42 2024-02-10T01:21:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_42 128M 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_42 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2024-02-10T01:21:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_42 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-02-10T01:21:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:21:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-02-10T01:21:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_42': 2024-02-10T01:21:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:21:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:21:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 146660de0e64 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.146660de0e64 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:03 2024 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:03 2024 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:03 2024 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae 2024-02-10T01:21:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:21:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:21:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_42 2024-02-10T01:21:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_42 2024-02-10T01:21:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:21:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=12 2024-02-10T01:21:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:21:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:21:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 7009280 bytes 2024-02-10T01:21:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 27038720 bytes 2024-02-10T01:21:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 40609792 bytes 2024-02-10T01:21:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 59539456 bytes 2024-02-10T01:21:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 106825728 bytes 2024-02-10T01:21:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 141801472 bytes 2024-02-10T01:21:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 202730496 bytes 2024-02-10T01:21:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 223025152 bytes 2024-02-10T01:21:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 316797952 bytes 2024-02-10T01:21:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 355173376 bytes 2024-02-10T01:21:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 371297280 bytes 2024-02-10T01:21:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 456680448 bytes 2024-02-10T01:21:06.522 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:21:06.522 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:21:06.522 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:21:06.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:21:06.523 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:21:06.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:21:06.526 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-02-10T01:21:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 472896512 bytes 2024-02-10T01:21:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 550974464 bytes 2024-02-10T01:21:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 579230720 bytes 2024-02-10T01:21:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 588334080 bytes 2024-02-10T01:21:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 622055424 bytes 2024-02-10T01:21:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 726453248 bytes 2024-02-10T01:21:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 749580288 bytes 2024-02-10T01:21:07.608 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-02-10T01:21:07.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-02-10T01:21:07.608 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-02-10T01:21:07.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-02-10T01:21:07.614 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-02-10T01:21:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 829790208 bytes 2024-02-10T01:21:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 837123072 bytes 2024-02-10T01:21:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 881556480 bytes 2024-02-10T01:21:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 915474432 bytes 2024-02-10T01:21:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 933896192 bytes 2024-02-10T01:21:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 948819968 bytes 2024-02-10T01:21:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 973971456 bytes 2024-02-10T01:21:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 998558720 bytes 2024-02-10T01:21:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1024063488 bytes 2024-02-10T01:21:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1041573888 bytes 2024-02-10T01:21:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1102112768 bytes 2024-02-10T01:21:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1149097984 bytes 2024-02-10T01:21:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1269847040 bytes 2024-02-10T01:21:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1347903488 bytes 2024-02-10T01:21:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1373810688 bytes 2024-02-10T01:21:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1447829504 bytes 2024-02-10T01:21:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1523257344 bytes 2024-02-10T01:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1530940416 bytes 2024-02-10T01:21:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1548943360 bytes 2024-02-10T01:21:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1622086656 bytes 2024-02-10T01:21:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1649328128 bytes 2024-02-10T01:21:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1691057152 bytes 2024-02-10T01:21:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1715126272 bytes 2024-02-10T01:21:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1785927680 bytes 2024-02-10T01:21:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1827497984 bytes 2024-02-10T01:21:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1870349312 bytes 2024-02-10T01:21:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1898681344 bytes 2024-02-10T01:21:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1939030016 bytes 2024-02-10T01:21:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2041085952 bytes 2024-02-10T01:21:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2062867456 bytes 2024-02-10T01:21:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2121307136 bytes 2024-02-10T01:21:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2173789184 bytes 2024-02-10T01:21:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2193581056 bytes 2024-02-10T01:21:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2261262336 bytes 2024-02-10T01:21:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2291203072 bytes 2024-02-10T01:21:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2342120448 bytes 2024-02-10T01:21:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2372503552 bytes 2024-02-10T01:21:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2393257984 bytes 2024-02-10T01:21:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2469429248 bytes 2024-02-10T01:21:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2490684416 bytes 2024-02-10T01:21:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2530853888 bytes 2024-02-10T01:21:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2549532672 bytes 2024-02-10T01:21:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2565832704 bytes 2024-02-10T01:21:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2612962304 bytes 2024-02-10T01:21:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2658651136 bytes 2024-02-10T01:21:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 2749966336 bytes 2024-02-10T01:21:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 2753949696 bytes 2024-02-10T01:21:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 2754569216 bytes 2024-02-10T01:21:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 2814704640 bytes 2024-02-10T01:21:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 2820458496 bytes 2024-02-10T01:21:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 2939013120 bytes 2024-02-10T01:21:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7100 IOs, 2974741504 bytes 2024-02-10T01:21:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7200 IOs, 3014962176 bytes 2024-02-10T01:21:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7300 IOs, 3072528384 bytes 2024-02-10T01:21:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7400 IOs, 3117668352 bytes 2024-02-10T01:21:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7500 IOs, 3118279680 bytes 2024-02-10T01:21:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7600 IOs, 3202365440 bytes 2024-02-10T01:21:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7700 IOs, 3231062016 bytes 2024-02-10T01:21:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7800 IOs, 3282158592 bytes 2024-02-10T01:21:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7900 IOs, 3329632256 bytes 2024-02-10T01:21:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8000 IOs, 3349488640 bytes 2024-02-10T01:21:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8100 IOs, 3403249664 bytes 2024-02-10T01:21:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8200 IOs, 3463482368 bytes 2024-02-10T01:21:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:8300 IOs, 3474509824 bytes 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_43 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_43 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_43 128M 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_43 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2024-02-10T01:21:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_43 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-02-10T01:21:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:21:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_43': 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1481141f553 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1481141f553 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:21:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:16 2024 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:16 2024 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:16 2024 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb 2024-02-10T01:21:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:21:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:21:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_43 2024-02-10T01:21:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_43 2024-02-10T01:21:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:21:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 76142592 bytes 2024-02-10T01:21:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 130344960 bytes 2024-02-10T01:21:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 172259328 bytes 2024-02-10T01:21:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 239537152 bytes 2024-02-10T01:21:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 281344000 bytes 2024-02-10T01:21:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 287145984 bytes 2024-02-10T01:21:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 291770368 bytes 2024-02-10T01:21:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 341954560 bytes 2024-02-10T01:21:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 436456448 bytes 2024-02-10T01:21:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 457088000 bytes 2024-02-10T01:21:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 461417472 bytes 2024-02-10T01:21:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 526471168 bytes 2024-02-10T01:21:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 566152192 bytes 2024-02-10T01:21:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 594872320 bytes 2024-02-10T01:21:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 648578048 bytes 2024-02-10T01:21:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 683687936 bytes 2024-02-10T01:21:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 731474944 bytes 2024-02-10T01:21:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 774328320 bytes 2024-02-10T01:21:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 860942336 bytes 2024-02-10T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 884779008 bytes 2024-02-10T01:21:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 964477952 bytes 2024-02-10T01:21:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1045929984 bytes 2024-02-10T01:21:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1152841728 bytes 2024-02-10T01:21:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1183237120 bytes 2024-02-10T01:21:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1256567808 bytes 2024-02-10T01:21:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1302449152 bytes 2024-02-10T01:21:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1337911296 bytes 2024-02-10T01:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1379070976 bytes 2024-02-10T01:21:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1405709312 bytes 2024-02-10T01:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1455143936 bytes 2024-02-10T01:21:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1470137344 bytes 2024-02-10T01:21:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1518242816 bytes 2024-02-10T01:21:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1556914176 bytes 2024-02-10T01:21:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1665814528 bytes 2024-02-10T01:21:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1720861696 bytes 2024-02-10T01:21:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1762774016 bytes 2024-02-10T01:21:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1818211328 bytes 2024-02-10T01:21:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1830683648 bytes 2024-02-10T01:21:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1854908416 bytes 2024-02-10T01:21:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1905905664 bytes 2024-02-10T01:21:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1918761984 bytes 2024-02-10T01:21:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1945800704 bytes 2024-02-10T01:21:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2001291264 bytes 2024-02-10T01:21:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2037338112 bytes 2024-02-10T01:21:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2037957632 bytes 2024-02-10T01:21:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2102109184 bytes 2024-02-10T01:21:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2102747136 bytes 2024-02-10T01:21:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2143837184 bytes 2024-02-10T01:21:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2213660672 bytes 2024-02-10T01:21:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_44 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_44 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_44 128M 2024-02-10T01:21:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_44 2024-02-10T01:21:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2024-02-10T01:21:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_44 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-02-10T01:21:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:21:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-02-10T01:21:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_44': 2024-02-10T01:21:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:21:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 148a82354126 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.148a82354126 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:23 2024 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:23 2024 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:23 2024 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:21:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: acaf3e70-c7f2-4137-b839-6ab0448165bb 2024-02-10T01:21:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:21:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:21:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_44 2024-02-10T01:21:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_44 2024-02-10T01:21:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:21:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=7 2024-02-10T01:21:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:21:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:21:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 21456896 bytes 2024-02-10T01:21:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 41537536 bytes 2024-02-10T01:21:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 75567104 bytes 2024-02-10T01:21:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 110038016 bytes 2024-02-10T01:21:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 117095424 bytes 2024-02-10T01:21:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 165971968 bytes 2024-02-10T01:21:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 198333440 bytes 2024-02-10T01:21:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 254426112 bytes 2024-02-10T01:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 288340992 bytes 2024-02-10T01:21:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 358961152 bytes 2024-02-10T01:21:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 402778112 bytes 2024-02-10T01:21:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 424047616 bytes 2024-02-10T01:21:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 466530304 bytes 2024-02-10T01:21:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 500796416 bytes 2024-02-10T01:21:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 529420288 bytes 2024-02-10T01:21:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 606455808 bytes 2024-02-10T01:21:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 630802432 bytes 2024-02-10T01:21:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 676313088 bytes 2024-02-10T01:21:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 736077824 bytes 2024-02-10T01:21:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 754400256 bytes 2024-02-10T01:21:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 788518912 bytes 2024-02-10T01:21:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 789120000 bytes 2024-02-10T01:21:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 836926464 bytes 2024-02-10T01:21:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 858562560 bytes 2024-02-10T01:21:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 901959680 bytes 2024-02-10T01:21:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 967991296 bytes 2024-02-10T01:21:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1021820928 bytes 2024-02-10T01:21:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1085300736 bytes 2024-02-10T01:21:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1167065088 bytes 2024-02-10T01:21:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1220373504 bytes 2024-02-10T01:21:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1243019264 bytes 2024-02-10T01:21:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1298929664 bytes 2024-02-10T01:21:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1351920640 bytes 2024-02-10T01:21:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1414422528 bytes 2024-02-10T01:21:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1473268736 bytes 2024-02-10T01:21:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1514228736 bytes 2024-02-10T01:21:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1588613120 bytes 2024-02-10T01:21:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1630641152 bytes 2024-02-10T01:21:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1642479616 bytes 2024-02-10T01:21:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1658221568 bytes 2024-02-10T01:21:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1740800000 bytes 2024-02-10T01:21:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1769220096 bytes 2024-02-10T01:21:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1835936768 bytes 2024-02-10T01:21:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1867259904 bytes 2024-02-10T01:21:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1929852928 bytes 2024-02-10T01:21:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1964612608 bytes 2024-02-10T01:21:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1995708416 bytes 2024-02-10T01:21:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2041532416 bytes 2024-02-10T01:21:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2089383936 bytes 2024-02-10T01:21:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2150090752 bytes 2024-02-10T01:21:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_45 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_45 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_45 128M 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_45 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_45 2024-02-10T01:21:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-02-10T01:21:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:21:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-02-10T01:21:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_45': 2024-02-10T01:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1492b4ee4f90 2024-02-10T01:21:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1492b4ee4f90 2024-02-10T01:21:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:21:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:21:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:21:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:21:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:31 2024 2024-02-10T01:21:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:31 2024 2024-02-10T01:21:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:31 2024 2024-02-10T01:21:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:21:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:21:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e5c1b9f4-c50d-41b1-be2c-c5c92fd22119 2024-02-10T01:21:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:21:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:21:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_45 2024-02-10T01:21:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_45 2024-02-10T01:21:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:21:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:21:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:21:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:21:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 617472 bytes 2024-02-10T01:21:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 26776576 bytes 2024-02-10T01:21:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 74332160 bytes 2024-02-10T01:21:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 148459520 bytes 2024-02-10T01:21:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 199143424 bytes 2024-02-10T01:21:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 266277888 bytes 2024-02-10T01:21:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 282091520 bytes 2024-02-10T01:21:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 348994560 bytes 2024-02-10T01:21:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 386644992 bytes 2024-02-10T01:21:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 387282944 bytes 2024-02-10T01:21:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 491342848 bytes 2024-02-10T01:21:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 521958400 bytes 2024-02-10T01:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 543636480 bytes 2024-02-10T01:21:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 606191616 bytes 2024-02-10T01:21:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 634631168 bytes 2024-02-10T01:21:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 654864384 bytes 2024-02-10T01:21:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 694217728 bytes 2024-02-10T01:21:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 750222336 bytes 2024-02-10T01:21:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 766526464 bytes 2024-02-10T01:21:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 823434240 bytes 2024-02-10T01:21:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 849565696 bytes 2024-02-10T01:21:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 914045952 bytes 2024-02-10T01:21:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 966951936 bytes 2024-02-10T01:21:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1023364096 bytes 2024-02-10T01:21:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1071300608 bytes 2024-02-10T01:21:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1071924224 bytes 2024-02-10T01:21:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1118700544 bytes 2024-02-10T01:21:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1149383680 bytes 2024-02-10T01:21:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1201619968 bytes 2024-02-10T01:21:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1218766848 bytes 2024-02-10T01:21:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1293521920 bytes 2024-02-10T01:21:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1302925312 bytes 2024-02-10T01:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1342564352 bytes 2024-02-10T01:21:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1374624768 bytes 2024-02-10T01:21:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1436368896 bytes 2024-02-10T01:21:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1438588928 bytes 2024-02-10T01:21:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1475371008 bytes 2024-02-10T01:21:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1523336192 bytes 2024-02-10T01:21:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1550142464 bytes 2024-02-10T01:21:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1575563264 bytes 2024-02-10T01:21:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1614761984 bytes 2024-02-10T01:21:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1650055168 bytes 2024-02-10T01:21:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1750810624 bytes 2024-02-10T01:21:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1811420160 bytes 2024-02-10T01:21:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1843299328 bytes 2024-02-10T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1867968512 bytes 2024-02-10T01:21:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1898122240 bytes 2024-02-10T01:21:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1947380736 bytes 2024-02-10T01:21:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2027256832 bytes 2024-02-10T01:21:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2048009216 bytes 2024-02-10T01:21:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2081995776 bytes 2024-02-10T01:21:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2108751872 bytes 2024-02-10T01:21:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2117594112 bytes 2024-02-10T01:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2180100096 bytes 2024-02-10T01:21:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2203101184 bytes 2024-02-10T01:21:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2343878656 bytes 2024-02-10T01:21:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2403875840 bytes 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_46 2024-02-10T01:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_46 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_46 128M 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_46 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_46 2024-02-10T01:21:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-02-10T01:21:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:21:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_46': 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 149af9f1b628 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.149af9f1b628 2024-02-10T01:21:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:39 2024 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:39 2024 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:39 2024 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0b89dccc-dd99-4059-94f0-74a8c09ad67b 2024-02-10T01:21:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:21:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:21:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_46 2024-02-10T01:21:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_46 2024-02-10T01:21:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:21:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=8 2024-02-10T01:21:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:21:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:21:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 31177728 bytes 2024-02-10T01:21:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 35590144 bytes 2024-02-10T01:21:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 73961472 bytes 2024-02-10T01:21:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 88117248 bytes 2024-02-10T01:21:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 110692352 bytes 2024-02-10T01:21:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 203142144 bytes 2024-02-10T01:21:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 217616384 bytes 2024-02-10T01:21:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 221528064 bytes 2024-02-10T01:21:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 281726976 bytes 2024-02-10T01:21:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 317431808 bytes 2024-02-10T01:21:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 362901504 bytes 2024-02-10T01:21:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 434522112 bytes 2024-02-10T01:21:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 468820992 bytes 2024-02-10T01:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 485800960 bytes 2024-02-10T01:21:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 577480704 bytes 2024-02-10T01:21:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 671707136 bytes 2024-02-10T01:21:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 708667392 bytes 2024-02-10T01:21:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 709236736 bytes 2024-02-10T01:21:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 731031552 bytes 2024-02-10T01:21:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 778104832 bytes 2024-02-10T01:21:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 824357888 bytes 2024-02-10T01:21:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 864322560 bytes 2024-02-10T01:21:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 891054080 bytes 2024-02-10T01:21:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 981706752 bytes 2024-02-10T01:21:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1061443584 bytes 2024-02-10T01:21:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1139030016 bytes 2024-02-10T01:21:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1246370816 bytes 2024-02-10T01:21:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1302684672 bytes 2024-02-10T01:21:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1330134016 bytes 2024-02-10T01:21:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1398322176 bytes 2024-02-10T01:21:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1422841856 bytes 2024-02-10T01:21:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1495403520 bytes 2024-02-10T01:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1498716160 bytes 2024-02-10T01:21:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1536112640 bytes 2024-02-10T01:21:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1608173568 bytes 2024-02-10T01:21:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1630110720 bytes 2024-02-10T01:21:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1676496896 bytes 2024-02-10T01:21:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1700068352 bytes 2024-02-10T01:21:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1725893632 bytes 2024-02-10T01:21:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1779785728 bytes 2024-02-10T01:21:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1837617152 bytes 2024-02-10T01:21:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1935685632 bytes 2024-02-10T01:21:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2003663872 bytes 2024-02-10T01:21:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2063974400 bytes 2024-02-10T01:21:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2150520832 bytes 2024-02-10T01:21:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2194659328 bytes 2024-02-10T01:21:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2243829760 bytes 2024-02-10T01:21:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2301478912 bytes 2024-02-10T01:21:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2326021120 bytes 2024-02-10T01:21:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2359371776 bytes 2024-02-10T01:21:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2426105856 bytes 2024-02-10T01:21:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2444817408 bytes 2024-02-10T01:21:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2468872192 bytes 2024-02-10T01:21:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2539136000 bytes 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_47 2024-02-10T01:21:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_47 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_47 128M 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_47 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_47 2024-02-10T01:21:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-02-10T01:21:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:21:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-02-10T01:21:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_47': 2024-02-10T01:21:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:21:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:21:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:21:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 14a471a81835 2024-02-10T01:21:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.14a471a81835 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:48 2024 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:48 2024 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:48 2024 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9adfb392-ed84-47b2-9257-a61dc69d11f1 2024-02-10T01:21:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:21:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:21:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_47 2024-02-10T01:21:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_47 2024-02-10T01:21:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:21:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=10 2024-02-10T01:21:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:21:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:21:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 60219392 bytes 2024-02-10T01:21:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 140796928 bytes 2024-02-10T01:21:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 191375360 bytes 2024-02-10T01:21:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 245941248 bytes 2024-02-10T01:21:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 285796352 bytes 2024-02-10T01:21:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 333322240 bytes 2024-02-10T01:21:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 407955456 bytes 2024-02-10T01:21:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 477593600 bytes 2024-02-10T01:21:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 524318720 bytes 2024-02-10T01:21:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 563392512 bytes 2024-02-10T01:21:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 587717632 bytes 2024-02-10T01:21:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 612106240 bytes 2024-02-10T01:21:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 652928000 bytes 2024-02-10T01:21:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 677722112 bytes 2024-02-10T01:21:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 704278528 bytes 2024-02-10T01:21:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 723484672 bytes 2024-02-10T01:21:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 747062272 bytes 2024-02-10T01:21:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 768215040 bytes 2024-02-10T01:21:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 822250496 bytes 2024-02-10T01:21:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 907247616 bytes 2024-02-10T01:21:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 952520704 bytes 2024-02-10T01:21:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 998588416 bytes 2024-02-10T01:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1030877184 bytes 2024-02-10T01:21:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1033203712 bytes 2024-02-10T01:21:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1111302144 bytes 2024-02-10T01:21:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1131843584 bytes 2024-02-10T01:21:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1176177664 bytes 2024-02-10T01:21:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1209930752 bytes 2024-02-10T01:21:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1325489152 bytes 2024-02-10T01:21:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1381096448 bytes 2024-02-10T01:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1420940288 bytes 2024-02-10T01:21:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1440937984 bytes 2024-02-10T01:21:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1490896896 bytes 2024-02-10T01:21:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1531948032 bytes 2024-02-10T01:21:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1587276800 bytes 2024-02-10T01:21:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1649286144 bytes 2024-02-10T01:21:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1708348416 bytes 2024-02-10T01:21:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1755401216 bytes 2024-02-10T01:21:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1777189888 bytes 2024-02-10T01:21:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1837379584 bytes 2024-02-10T01:21:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1922050048 bytes 2024-02-10T01:21:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1984796672 bytes 2024-02-10T01:21:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2016905216 bytes 2024-02-10T01:21:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2052165632 bytes 2024-02-10T01:21:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2069836800 bytes 2024-02-10T01:21:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2114317312 bytes 2024-02-10T01:21:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2210075648 bytes 2024-02-10T01:21:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2280409088 bytes 2024-02-10T01:21:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2314990592 bytes 2024-02-10T01:21:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2367638528 bytes 2024-02-10T01:21:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2397457408 bytes 2024-02-10T01:21:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2464235520 bytes 2024-02-10T01:21:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2506966016 bytes 2024-02-10T01:21:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2544199680 bytes 2024-02-10T01:21:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2636596224 bytes 2024-02-10T01:21:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2659784704 bytes 2024-02-10T01:21:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2680015872 bytes 2024-02-10T01:21:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2697016320 bytes 2024-02-10T01:21:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2780865536 bytes 2024-02-10T01:21:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2807697408 bytes 2024-02-10T01:21:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2863665152 bytes 2024-02-10T01:21:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2899314688 bytes 2024-02-10T01:21:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2967367680 bytes 2024-02-10T01:21:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2983952384 bytes 2024-02-10T01:21:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6500 IOs, 3058769920 bytes 2024-02-10T01:21:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6600 IOs, 3116914688 bytes 2024-02-10T01:21:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6700 IOs, 3138620416 bytes 2024-02-10T01:21:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6800 IOs, 3154337792 bytes 2024-02-10T01:21:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6900 IOs, 3177806848 bytes 2024-02-10T01:21:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:7000 IOs, 3210565632 bytes 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_48 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_48 2024-02-10T01:21:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:21:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_48 128M 2024-02-10T01:21:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_48 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:21:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2024-02-10T01:21:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:21:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_48 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:21:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-02-10T01:22:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:22:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-02-10T01:22:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_48': 2024-02-10T01:22:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:22:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:22:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 14ac63eee2f0 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.14ac63eee2f0 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:59 2024 2024-02-10T01:22:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:59 2024 2024-02-10T01:22:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:59 2024 2024-02-10T01:22:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:22:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:22:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 397dba79-e127-495d-94c6-4a204e4c5269 2024-02-10T01:22:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:22:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:22:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_48 2024-02-10T01:22:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_48 2024-02-10T01:22:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:22:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=9 2024-02-10T01:22:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:22:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:22:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 48722944 bytes 2024-02-10T01:22:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 124208128 bytes 2024-02-10T01:22:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 140217344 bytes 2024-02-10T01:22:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 257910784 bytes 2024-02-10T01:22:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 287558656 bytes 2024-02-10T01:22:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 341106688 bytes 2024-02-10T01:22:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 341880832 bytes 2024-02-10T01:22:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 362404864 bytes 2024-02-10T01:22:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 413298688 bytes 2024-02-10T01:22:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 435698688 bytes 2024-02-10T01:22:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 445540352 bytes 2024-02-10T01:22:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 550444032 bytes 2024-02-10T01:22:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 595842048 bytes 2024-02-10T01:22:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 631440384 bytes 2024-02-10T01:22:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 703299584 bytes 2024-02-10T01:22:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 747394048 bytes 2024-02-10T01:22:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 795891712 bytes 2024-02-10T01:22:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 882272256 bytes 2024-02-10T01:22:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 887276544 bytes 2024-02-10T01:22:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 923498496 bytes 2024-02-10T01:22:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 992594944 bytes 2024-02-10T01:22:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1087387648 bytes 2024-02-10T01:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1136931840 bytes 2024-02-10T01:22:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1195699200 bytes 2024-02-10T01:22:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1196303360 bytes 2024-02-10T01:22:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1239782400 bytes 2024-02-10T01:22:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1279869952 bytes 2024-02-10T01:22:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1329609728 bytes 2024-02-10T01:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1357389824 bytes 2024-02-10T01:22:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1399662592 bytes 2024-02-10T01:22:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1409625088 bytes 2024-02-10T01:22:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1419397120 bytes 2024-02-10T01:22:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1420004352 bytes 2024-02-10T01:22:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1432834048 bytes 2024-02-10T01:22:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1433979904 bytes 2024-02-10T01:22:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1459782656 bytes 2024-02-10T01:22:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1483518976 bytes 2024-02-10T01:22:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1540963328 bytes 2024-02-10T01:22:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1558061056 bytes 2024-02-10T01:22:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1565005824 bytes 2024-02-10T01:22:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1662341120 bytes 2024-02-10T01:22:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1679947776 bytes 2024-02-10T01:22:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1746363392 bytes 2024-02-10T01:22:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1756273664 bytes 2024-02-10T01:22:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1828911104 bytes 2024-02-10T01:22:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1848982528 bytes 2024-02-10T01:22:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1874675712 bytes 2024-02-10T01:22:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1896656896 bytes 2024-02-10T01:22:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 1941584896 bytes 2024-02-10T01:22:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2031714304 bytes 2024-02-10T01:22:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2072541184 bytes 2024-02-10T01:22:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2097408000 bytes 2024-02-10T01:22:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2126025728 bytes 2024-02-10T01:22:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2215745536 bytes 2024-02-10T01:22:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2236935168 bytes 2024-02-10T01:22:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2276177920 bytes 2024-02-10T01:22:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2313046016 bytes 2024-02-10T01:22:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2362695680 bytes 2024-02-10T01:22:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2409915392 bytes 2024-02-10T01:22:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2489027584 bytes 2024-02-10T01:22:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2500704256 bytes 2024-02-10T01:22:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6200 IOs, 2611999744 bytes 2024-02-10T01:22:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6300 IOs, 2695825408 bytes 2024-02-10T01:22:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6400 IOs, 2759861248 bytes 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_49 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_49 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_49 128M 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_49 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:22:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:22:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:22:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2024-02-10T01:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-02-10T01:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_49 2024-02-10T01:22:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:22:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-02-10T01:22:10.527 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:22:10.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:22:10.527 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-02-10T01:22:10.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:22:10.527+0000 7f8b3ee38640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 51147) UID: 1000 2024-02-10T01:22:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:22:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_49': 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 14b49998859f 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.14b49998859f 2024-02-10T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:22:09 2024 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:22:09 2024 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:22:09 2024 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c13056e5-b191-4d8c-96f9-917ad3e45ceb 2024-02-10T01:22:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_49 2024-02-10T01:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_49 2024-02-10T01:22:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:22:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=11 2024-02-10T01:22:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:22:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:22:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 37490688 bytes 2024-02-10T01:22:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 43979776 bytes 2024-02-10T01:22:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 164907008 bytes 2024-02-10T01:22:11.528 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:22:11.528 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:22:11.528 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:22:11.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:22:11.529 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:22:11.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:22:11.531 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-02-10T01:22:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 241553408 bytes 2024-02-10T01:22:11.615 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5858267830005915) > weight (0.25) 2024-02-10T01:22:11.615 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4765977335754613) > weight (0.25) 2024-02-10T01:22:11.615 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9166504760575959) > weight (0.25) 2024-02-10T01:22:11.615 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.42032585289262303) > weight (0.25) 2024-02-10T01:22:11.616 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-02-10T01:22:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 302786560 bytes 2024-02-10T01:22:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 364721152 bytes 2024-02-10T01:22:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 389248000 bytes 2024-02-10T01:22:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 408501248 bytes 2024-02-10T01:22:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 443502592 bytes 2024-02-10T01:22:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 491530240 bytes 2024-02-10T01:22:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 523368448 bytes 2024-02-10T01:22:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 560827392 bytes 2024-02-10T01:22:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 598583296 bytes 2024-02-10T01:22:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 674884608 bytes 2024-02-10T01:22:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 717877248 bytes 2024-02-10T01:22:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 772917248 bytes 2024-02-10T01:22:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 796943360 bytes 2024-02-10T01:22:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 871424000 bytes 2024-02-10T01:22:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 944290816 bytes 2024-02-10T01:22:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 996244480 bytes 2024-02-10T01:22:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 1050332160 bytes 2024-02-10T01:22:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 1079154688 bytes 2024-02-10T01:22:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1093548032 bytes 2024-02-10T01:22:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1157053440 bytes 2024-02-10T01:22:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1219026944 bytes 2024-02-10T01:22:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1250268160 bytes 2024-02-10T01:22:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1342612480 bytes 2024-02-10T01:22:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1393028096 bytes 2024-02-10T01:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1406149632 bytes 2024-02-10T01:22:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1428966400 bytes 2024-02-10T01:22:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1529353216 bytes 2024-02-10T01:22:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1569425408 bytes 2024-02-10T01:22:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1572881408 bytes 2024-02-10T01:22:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1615464448 bytes 2024-02-10T01:22:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1635978240 bytes 2024-02-10T01:22:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1703413760 bytes 2024-02-10T01:22:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1790526464 bytes 2024-02-10T01:22:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1842782208 bytes 2024-02-10T01:22:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1922438144 bytes 2024-02-10T01:22:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 2008010752 bytes 2024-02-10T01:22:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 2047196160 bytes 2024-02-10T01:22:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 2105367552 bytes 2024-02-10T01:22:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 2144031744 bytes 2024-02-10T01:22:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 2215998464 bytes 2024-02-10T01:22:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 2286884864 bytes 2024-02-10T01:22:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 2341131264 bytes 2024-02-10T01:22:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 2377113600 bytes 2024-02-10T01:22:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 2398148608 bytes 2024-02-10T01:22:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2431722496 bytes 2024-02-10T01:22:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5000 IOs, 2455319552 bytes 2024-02-10T01:22:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5100 IOs, 2498272256 bytes 2024-02-10T01:22:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5200 IOs, 2514179072 bytes 2024-02-10T01:22:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5300 IOs, 2546826240 bytes 2024-02-10T01:22:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5400 IOs, 2620978176 bytes 2024-02-10T01:22:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5500 IOs, 2666601472 bytes 2024-02-10T01:22:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5600 IOs, 2708700160 bytes 2024-02-10T01:22:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5700 IOs, 2776616960 bytes 2024-02-10T01:22:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5800 IOs, 2801203200 bytes 2024-02-10T01:22:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:5900 IOs, 2886274048 bytes 2024-02-10T01:22:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6000 IOs, 2921228288 bytes 2024-02-10T01:22:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:6100 IOs, 2957658112 bytes 2024-02-10T01:22:19.903 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:22:19.901+0000 7febc4353640 -1 bdev(0x561eb8f3ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7febc4353640 time 2024-02-10T01:22:19.902531+0000 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x561eb56791b8] 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:*** Caught signal (Aborted) ** 2024-02-10T01:22:19.904 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: in thread 7febc4353640 thread_name:bstore_aio 2024-02-10T01:22:19.905 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.905 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7febd1254db0] 2024-02-10T01:22:19.905 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7febd12a154c] 2024-02-10T01:22:19.905 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: raise() 2024-02-10T01:22:19.905 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: abort() 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x561eb5679279] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 7: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:22:19.903+0000 7febc4353640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7febc4353640 time 2024-02-10T01:22:19.902531+0000 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x561eb56791b8] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.906 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-02-10T01:22:19.904+0000 7febc4353640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: in thread 7febc4353640 thread_name:bstore_aio 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7febd1254db0] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7febd12a154c] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: raise() 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: abort() 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x561eb5679279] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 7: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.907 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.908 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:22:19.908 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.924 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: -105> 2024-02-10T01:22:19.901+0000 7febc4353640 -1 bdev(0x561eb8f3ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: -26> 2024-02-10T01:22:19.903+0000 7febc4353640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7febc4353640 time 2024-02-10T01:22:19.902531+0000 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x561eb56791b8] 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 0> 2024-02-10T01:22:19.904+0000 7febc4353640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: in thread 7febc4353640 thread_name:bstore_aio 2024-02-10T01:22:19.925 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7febd1254db0] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7febd12a154c] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: raise() 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: abort() 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x561eb5679279] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 7: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:22:19.926 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.936 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: -474> 2024-02-10T01:22:19.901+0000 7febc4353640 -1 bdev(0x561eb8f3ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-02-10T01:22:19.936 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: -395> 2024-02-10T01:22:19.903+0000 7febc4353640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7febc4353640 time 2024-02-10T01:22:19.902531+0000 2024-02-10T01:22:19.936 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x561eb56791b8] 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: -369> 2024-02-10T01:22:19.904+0000 7febc4353640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: in thread 7febc4353640 thread_name:bstore_aio 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:22:19.937 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7febd1254db0] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7febd12a154c] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 3: raise() 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 4: abort() 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x561eb5679279] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x561eb6040110] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 7: ceph-osd(+0xdb8281) [0x561eb6040281] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7febd129f802] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7febd123f450] 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:22:19.938 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr: 2024-02-10T01:22:20.106 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:daemon-helper: command failed with exit status 5 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_50 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-02-10T01:22:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_50 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster2 mirror image_50 128M 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_50 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 128M ']' 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=128M 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-02-10T01:22:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2024-02-10T01:22:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-02-10T01:22:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_50 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-02-10T01:22:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-02-10T01:22:22.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-02-10T01:22:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-02-10T01:22:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-02-10T01:22:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_50': 2024-02-10T01:22:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:22:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 14c3b872a1ae 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.14c3b872a1ae 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:22:21 2024 2024-02-10T01:22:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:22:21 2024 2024-02-10T01:22:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:22:21 2024 2024-02-10T01:22:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:22:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:22:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 042fec75-49d5-40e6-98a0-d964d373c5d4 2024-02-10T01:22:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-02-10T01:22:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-02-10T01:22:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stress_write_image cluster2 mirror image_50 2024-02-10T01:22:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_50 2024-02-10T01:22:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-02-10T01:22:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local duration=5 2024-02-10T01:22:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:22:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-02-10T01:22:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:100 IOs, 35993600 bytes 2024-02-10T01:22:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:200 IOs, 42912768 bytes 2024-02-10T01:22:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:300 IOs, 119641088 bytes 2024-02-10T01:22:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:400 IOs, 124127232 bytes 2024-02-10T01:22:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:500 IOs, 203510784 bytes 2024-02-10T01:22:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:600 IOs, 270241792 bytes 2024-02-10T01:22:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:700 IOs, 310893568 bytes 2024-02-10T01:22:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:800 IOs, 323026944 bytes 2024-02-10T01:22:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:900 IOs, 358293504 bytes 2024-02-10T01:22:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1000 IOs, 455426048 bytes 2024-02-10T01:22:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1100 IOs, 479838208 bytes 2024-02-10T01:22:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1200 IOs, 535366656 bytes 2024-02-10T01:22:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1300 IOs, 549791744 bytes 2024-02-10T01:22:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1400 IOs, 592711680 bytes 2024-02-10T01:22:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1500 IOs, 642588672 bytes 2024-02-10T01:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1600 IOs, 705244160 bytes 2024-02-10T01:22:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1700 IOs, 724683776 bytes 2024-02-10T01:22:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1800 IOs, 751396864 bytes 2024-02-10T01:22:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:1900 IOs, 803601408 bytes 2024-02-10T01:22:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2000 IOs, 844725248 bytes 2024-02-10T01:22:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2100 IOs, 913435648 bytes 2024-02-10T01:22:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2200 IOs, 999642112 bytes 2024-02-10T01:22:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2300 IOs, 1013829632 bytes 2024-02-10T01:22:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2400 IOs, 1057028096 bytes 2024-02-10T01:22:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2500 IOs, 1093824512 bytes 2024-02-10T01:22:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2600 IOs, 1110533120 bytes 2024-02-10T01:22:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2700 IOs, 1143815168 bytes 2024-02-10T01:22:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2800 IOs, 1189253120 bytes 2024-02-10T01:22:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:2900 IOs, 1197918208 bytes 2024-02-10T01:22:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3000 IOs, 1216384000 bytes 2024-02-10T01:22:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3100 IOs, 1236208640 bytes 2024-02-10T01:22:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3200 IOs, 1296478208 bytes 2024-02-10T01:22:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3300 IOs, 1340832768 bytes 2024-02-10T01:22:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3400 IOs, 1415144448 bytes 2024-02-10T01:22:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3500 IOs, 1465203712 bytes 2024-02-10T01:22:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3600 IOs, 1466477568 bytes 2024-02-10T01:22:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3700 IOs, 1534136320 bytes 2024-02-10T01:22:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3800 IOs, 1589656576 bytes 2024-02-10T01:22:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:3900 IOs, 1622767616 bytes 2024-02-10T01:22:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4000 IOs, 1654409216 bytes 2024-02-10T01:22:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4100 IOs, 1677859840 bytes 2024-02-10T01:22:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4200 IOs, 1690320896 bytes 2024-02-10T01:22:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4300 IOs, 1772303360 bytes 2024-02-10T01:22:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4400 IOs, 1831501824 bytes 2024-02-10T01:22:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4500 IOs, 1864378368 bytes 2024-02-10T01:22:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4600 IOs, 1899471872 bytes 2024-02-10T01:22:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4700 IOs, 1958820864 bytes 2024-02-10T01:22:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4800 IOs, 1985789952 bytes 2024-02-10T01:22:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:4900 IOs, 2019211264 bytes 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=124 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 124 -eq 124 ']' 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image_count=50 2024-02-10T01:22:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:22:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local count 2024-02-10T01:22:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_count=0 2024-02-10T01:22:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:22:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 40 2024-02-10T01:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in `seq 1 40` 2024-02-10T01:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 1 -ne 1 2024-02-10T01:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-02-10T01:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep 'images: ' 2024-02-10T01:22:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cut '-d ' -f 2 2024-02-10T01:22:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ count=50 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 50 = 50 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state 2024-02-10T01:22:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 40 2024-02-10T01:22:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in `seq 1 40` 2024-02-10T01:22:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 1 -ne 1 2024-02-10T01:22:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-02-10T01:22:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep 'image health:' 2024-02-10T01:22:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cut '-d ' -f 3 2024-02-10T01:22:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:22:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=WARNING 2024-02-10T01:22:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test WARNING = ERROR 2024-02-10T01:22:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test WARNING = OK 2024-02-10T01:22:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in `seq 1 40` 2024-02-10T01:22:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 2 -ne 1 2024-02-10T01:22:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 30 2024-02-10T01:22:28.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-02-10T01:22:33.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2024-02-10T01:22:39.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2024-02-10T01:22:44.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2024-02-10T01:22:49.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2024-02-10T01:22:55.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2024-02-10T01:22:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-02-10T01:22:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep 'image health:' 2024-02-10T01:22:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cut '-d ' -f 3 2024-02-10T01:22:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=OK 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test OK = ERROR 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test OK = OK 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image_count=50 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local count 2024-02-10T01:22:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_count=0 2024-02-10T01:22:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:22:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 40 2024-02-10T01:22:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in `seq 1 40` 2024-02-10T01:22:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 1 -ne 1 2024-02-10T01:22:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-02-10T01:22:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep 'images: ' 2024-02-10T01:22:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cut '-d ' -f 2 2024-02-10T01:22:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ count=50 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 50 = 50 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state 2024-02-10T01:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 40 2024-02-10T01:22:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in `seq 1 40` 2024-02-10T01:22:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 1 -ne 1 2024-02-10T01:22:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-02-10T01:22:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep 'image health:' 2024-02-10T01:22:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cut '-d ' -f 3 2024-02-10T01:22:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: failed to query services: (13) Permission denied 2024-02-10T01:22:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=OK 2024-02-10T01:22:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test OK = ERROR 2024-02-10T01:22:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test OK = OK 2024-02-10T01:22:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:22:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ testlog 'TEST: compare many images' 2024-02-10T01:22:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ tee -a /tmp/tmp.ywJfcqGoUP/rbd-mirror.test.log 2024-02-10T01:22:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ date '+%F %T' 2024-02-10T01:22:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10 01:22:58 TEST: compare many images 2024-02-10T01:22:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10 01:22:58 TEST: compare many images 2024-02-10T01:22:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 1 50 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_1 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_1 snap 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:22:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-02-10T01:22:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:22:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:22:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-02-10T01:22:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:22:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:22:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:22:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:22:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:22:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:22:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-02-10T01:22:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:22:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:22:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:22:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:22:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:22:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:22:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:22:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:22:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:22:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:22:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:22:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-02-10T01:22:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:22:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:22:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:22:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:22:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:22:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:22:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:22:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:22:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:22:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:22:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-02-10T01:23:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:23:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:23:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:23:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:23:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:23:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_1.status 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_1.status 2024-02-10T01:23:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.status 2024-02-10T01:23:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_1.status 2024-02-10T01:23:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_1.status 2024-02-10T01:23:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-02-10T01:23:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-02-10T01:23:00.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2024-02-10T01:23:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 87 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.status 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.status 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-02-10T01:23:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:23:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=87 2024-02-10T01:23:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-02-10T01:23:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 87 = '' 2024-02-10T01:23:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-02-10T01:23:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 87 = '' 2024-02-10T01:23:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.status 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-02-10T01:23:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=87 2024-02-10T01:23:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 87 = 87 2024-02-10T01:23:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:23:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_1': 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10adb729b73c 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10adb729b73c 2024-02-10T01:23:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:07 2024 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:07 2024 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:07 2024 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:23:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_1 snap 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_1 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.export 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.export 2024-02-10T01:23:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.export 2024-02-10T01:23:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.export 2024-02-10T01:23:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.export 2024-02-10T01:23:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.export 2024-02-10T01:23:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_1.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_1.export 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_2 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_2 snap 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:23:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-02-10T01:23:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:23:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:23:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-02-10T01:23:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:no valid command found; 1 closest matches: 2024-02-10T01:23:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_20 2024-02-10T01:23:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-02-10T01:23:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:no valid command found; 5 closest matches: 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_22 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_23 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_26 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_27 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_28 2024-02-10T01:23:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-02-10T01:23:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:no valid command found; 3 closest matches: 2024-02-10T01:23:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_21 2024-02-10T01:23:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_25 2024-02-10T01:23:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_29 2024-02-10T01:23:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-02-10T01:23:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-02-10T01:23:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:23:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:23:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:23:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:23:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:23:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:23:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:23:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_2.status 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_2.status 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.status 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_2.status 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_2.status 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-02-10T01:23:06.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2024-02-10T01:23:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 89 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.status 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.status 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-02-10T01:23:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:23:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=89 2024-02-10T01:23:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:23:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-02-10T01:23:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=37 2024-02-10T01:23:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 89 = 37 2024-02-10T01:23:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:23:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-02-10T01:23:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 89 = '' 2024-02-10T01:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-02-10T01:23:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 89 = '' 2024-02-10T01:23:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:23:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.status 2024-02-10T01:23:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-02-10T01:23:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=89 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 89 = 89 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:23:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_2': 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165e057a8a9 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165e057a8a9 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:23:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:14 2024 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:14 2024 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:14 2024 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:23:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_2 snap 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_2 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.export 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.export 2024-02-10T01:23:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.export 2024-02-10T01:23:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.export 2024-02-10T01:23:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.export 2024-02-10T01:23:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.export 2024-02-10T01:23:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_2.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_2.export 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_3 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_3 snap 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:23:11.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-02-10T01:23:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:23:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:23:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:23:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:23:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-02-10T01:23:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-02-10T01:23:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:23:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:23:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:23:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:23:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_3.status 2024-02-10T01:23:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_3.status 2024-02-10T01:23:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.status 2024-02-10T01:23:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_3.status 2024-02-10T01:23:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_3.status 2024-02-10T01:23:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-02-10T01:23:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-02-10T01:23:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 91 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.status 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.status 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-02-10T01:23:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:23:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=91 2024-02-10T01:23:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:23:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-02-10T01:23:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 91 = '' 2024-02-10T01:23:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:23:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-02-10T01:23:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 91 = '' 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.status 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-02-10T01:23:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=91 2024-02-10T01:23:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 91 = 91 2024-02-10T01:23:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-02-10T01:23:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:23:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-02-10T01:23:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_3': 2024-02-10T01:23:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:23:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:23:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:23:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa200b883a 2024-02-10T01:23:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa200b883a 2024-02-10T01:23:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:23 2024 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:23 2024 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:23 2024 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:23:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:23:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_3 snap 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_3 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.export 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.export 2024-02-10T01:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.export 2024-02-10T01:23:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.export 2024-02-10T01:23:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.export 2024-02-10T01:23:16.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2024-02-10T01:23:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.export 2024-02-10T01:23:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_3.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_3.export 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_4 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_4 snap 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:23:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-02-10T01:23:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:no valid command found; 2 closest matches: 2024-02-10T01:23:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_44 2024-02-10T01:23:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status mirror/image_48 2024-02-10T01:23:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-02-10T01:23:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:23:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:23:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:23:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:23:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_4.status 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_4.status 2024-02-10T01:23:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.status 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_4.status 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_4.status 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-02-10T01:23:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-02-10T01:23:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 93 2024-02-10T01:23:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.status 2024-02-10T01:23:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.status 2024-02-10T01:23:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-02-10T01:23:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:23:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=93 2024-02-10T01:23:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-02-10T01:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 93 = '' 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:23:20.533 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:23:20.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:23:20.534 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-02-10T01:23:20.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:23:20.536+0000 7f82628d7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 51933) UID: 1000 2024-02-10T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-02-10T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:23:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:23:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:23:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:23:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 93 = '' 2024-02-10T01:23:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:23:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.status 2024-02-10T01:23:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-02-10T01:23:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=93 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 93 = 93 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:23:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:22.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2024-02-10T01:23:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_4': 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12012cb6b807 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12012cb6b807 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:23:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:31 2024 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:31 2024 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:31 2024 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:23:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:23:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_4 snap 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_4 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.export 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.export 2024-02-10T01:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.export 2024-02-10T01:23:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.export 2024-02-10T01:23:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.export 2024-02-10T01:23:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:23:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.export 2024-02-10T01:23:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_4.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_4.export 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_5 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_5 snap 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:23:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:23:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-02-10T01:23:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-02-10T01:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-02-10T01:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-02-10T01:23:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-02-10T01:23:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-02-10T01:23:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-02-10T01:23:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 1 2024-02-10T01:23:25.535 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:23:25.535 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:23:25.535 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:23:25.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:23:25.536 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:23:25.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:23:25.538 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-02-10T01:23:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-02-10T01:23:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-02-10T01:23:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-02-10T01:23:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-02-10T01:23:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-02-10T01:23:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 2 2024-02-10T01:23:27.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2024-02-10T01:23:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-02-10T01:23:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-02-10T01:23:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-02-10T01:23:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-02-10T01:23:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-02-10T01:23:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 4 2024-02-10T01:23:33.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2024-02-10T01:23:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-02-10T01:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-02-10T01:23:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-02-10T01:23:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-02-10T01:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 8 2024-02-10T01:23:38.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2024-02-10T01:23:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-02-10T01:23:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-02-10T01:23:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-02-10T01:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 8 2024-02-10T01:23:43.616 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5827771560096277) > weight (0.25) 2024-02-10T01:23:43.616 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-02-10T01:23:43.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-02-10T01:23:43.617 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-02-10T01:23:43.617 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-02-10T01:23:43.617 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:23:43.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi060.stderr:2024-02-10T01:23:43.617+0000 7f4cf84f1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 51254) UID: 1000 2024-02-10T01:23:43.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2024-02-10T01:23:49.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2024-02-10T01:23:49.719 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-02-10T01:23:49.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-02-10T01:23:49.719 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-02-10T01:23:49.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-02-10T01:23:49.723 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-02-10T01:23:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-02-10T01:23:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:23:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:23:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:23:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:23:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:23:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-02-10T01:23:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:23:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:23:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:23:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:23:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:23:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:23:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:23:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-02-10T01:23:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-02-10T01:23:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:23:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:23:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:23:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:23:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:23:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:23:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-02-10T01:23:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-02-10T01:23:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:23:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:23:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:23:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 8 2024-02-10T01:23:54.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2024-02-10T01:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:23:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:23:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:23:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:23:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-02-10T01:23:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:23:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:23:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:23:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:23:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:23:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:23:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:23:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_5.status 2024-02-10T01:23:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_5.status 2024-02-10T01:23:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.status 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_5.status 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_5.status 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-02-10T01:23:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 95 2024-02-10T01:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.status 2024-02-10T01:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.status 2024-02-10T01:23:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-02-10T01:23:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:23:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=95 2024-02-10T01:23:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:23:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-02-10T01:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:23:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:23:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=40 2024-02-10T01:23:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 95 = 40 2024-02-10T01:23:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:23:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:23:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:23:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:23:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:23:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:23:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:23:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-02-10T01:23:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=40 2024-02-10T01:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 95 = 40 2024-02-10T01:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:00.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2024-02-10T01:24:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:24:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:24:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:24:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-02-10T01:24:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 95 = '' 2024-02-10T01:24:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:24:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.status 2024-02-10T01:24:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-02-10T01:24:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=95 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 95 = 95 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-02-10T01:24:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_5': 2024-02-10T01:24:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10ad417595b7 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10ad417595b7 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:38 2024 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:38 2024 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:38 2024 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:24:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_5 snap 2024-02-10T01:24:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_5 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.export 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.export 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.export 2024-02-10T01:24:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.export 2024-02-10T01:24:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.export 2024-02-10T01:24:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.export 2024-02-10T01:24:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_5.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_5.export 2024-02-10T01:24:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_6 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_6 snap 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-02-10T01:24:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-02-10T01:24:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-02-10T01:24:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-02-10T01:24:05.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2024-02-10T01:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-02-10T01:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-02-10T01:24:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:24:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:24:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-02-10T01:24:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_6.status 2024-02-10T01:24:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_6.status 2024-02-10T01:24:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.status 2024-02-10T01:24:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_6.status 2024-02-10T01:24:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_6.status 2024-02-10T01:24:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-02-10T01:24:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-02-10T01:24:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 97 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.status 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.status 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-02-10T01:24:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=97 2024-02-10T01:24:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-02-10T01:24:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=41 2024-02-10T01:24:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 97 = 41 2024-02-10T01:24:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-02-10T01:24:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=41 2024-02-10T01:24:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 97 = 41 2024-02-10T01:24:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-02-10T01:24:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 97 = '' 2024-02-10T01:24:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.status 2024-02-10T01:24:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-02-10T01:24:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=97 2024-02-10T01:24:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 97 = 97 2024-02-10T01:24:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-02-10T01:24:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-02-10T01:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_6': 2024-02-10T01:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165a5908880 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165a5908880 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:43 2024 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:43 2024 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:43 2024 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:24:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_6 snap 2024-02-10T01:24:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_6 2024-02-10T01:24:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.export 2024-02-10T01:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.export 2024-02-10T01:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.export 2024-02-10T01:24:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.export 2024-02-10T01:24:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.export 2024-02-10T01:24:10.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2024-02-10T01:24:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.export 2024-02-10T01:24:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_6.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_6.export 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_7 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_7 snap 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-02-10T01:24:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-02-10T01:24:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-02-10T01:24:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-02-10T01:24:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-02-10T01:24:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-02-10T01:24:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:24:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:24:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:24:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-02-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-02-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:24:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:24:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2024-02-10T01:24:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_7.status 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_7.status 2024-02-10T01:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.status 2024-02-10T01:24:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_7.status 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_7.status 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-02-10T01:24:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 99 2024-02-10T01:24:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.status 2024-02-10T01:24:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.status 2024-02-10T01:24:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-02-10T01:24:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=99 2024-02-10T01:24:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-02-10T01:24:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=42 2024-02-10T01:24:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 99 = 42 2024-02-10T01:24:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-02-10T01:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=42 2024-02-10T01:24:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 99 = 42 2024-02-10T01:24:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-02-10T01:24:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 99 = '' 2024-02-10T01:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:16.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.status 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-02-10T01:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=99 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 99 = 99 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-02-10T01:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_7': 2024-02-10T01:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa12959935 2024-02-10T01:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa12959935 2024-02-10T01:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:50 2024 2024-02-10T01:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:50 2024 2024-02-10T01:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:50 2024 2024-02-10T01:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_7 snap 2024-02-10T01:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_7 2024-02-10T01:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.export 2024-02-10T01:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.export 2024-02-10T01:24:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.export 2024-02-10T01:24:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.export 2024-02-10T01:24:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.export 2024-02-10T01:24:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.export 2024-02-10T01:24:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_7.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_7.export 2024-02-10T01:24:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_8 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_8 snap 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-02-10T01:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-02-10T01:24:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-02-10T01:24:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-02-10T01:24:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-02-10T01:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_8.status 2024-02-10T01:24:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_8.status 2024-02-10T01:24:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.status 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_8.status 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_8.status 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-02-10T01:24:21.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2024-02-10T01:24:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 101 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.status 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.status 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-02-10T01:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=101 2024-02-10T01:24:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-02-10T01:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 101 = '' 2024-02-10T01:24:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-02-10T01:24:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 101 = '' 2024-02-10T01:24:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.status 2024-02-10T01:24:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-02-10T01:24:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=101 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 101 = 101 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-02-10T01:24:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_8': 2024-02-10T01:24:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201ba34cb1 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201ba34cb1 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:57 2024 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:14:57 2024 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:57 2024 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:24:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_8 snap 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_8 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.export 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.export 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.export 2024-02-10T01:24:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.export 2024-02-10T01:24:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.export 2024-02-10T01:24:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.export 2024-02-10T01:24:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_8.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_8.export 2024-02-10T01:24:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_9 2024-02-10T01:24:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_9 snap 2024-02-10T01:24:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-02-10T01:24:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_9.status 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_9.status 2024-02-10T01:24:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.status 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_9.status 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_9.status 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-02-10T01:24:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-02-10T01:24:27.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2024-02-10T01:24:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 103 2024-02-10T01:24:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.status 2024-02-10T01:24:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.status 2024-02-10T01:24:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-02-10T01:24:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=103 2024-02-10T01:24:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-02-10T01:24:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=44 2024-02-10T01:24:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 103 = 44 2024-02-10T01:24:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-02-10T01:24:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 103 = '' 2024-02-10T01:24:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-02-10T01:24:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 103 = '' 2024-02-10T01:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.status 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-02-10T01:24:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=103 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 103 = 103 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_9': 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10adb06ecad1 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10adb06ecad1 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:08 2024 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:08 2024 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:08 2024 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:24:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_9 snap 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_9 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.export 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.export 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.export 2024-02-10T01:24:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.export 2024-02-10T01:24:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.export 2024-02-10T01:24:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.export 2024-02-10T01:24:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_9.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_9.export 2024-02-10T01:24:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_10 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_10 snap 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:32.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2024-02-10T01:24:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-02-10T01:24:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-02-10T01:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-02-10T01:24:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-02-10T01:24:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-02-10T01:24:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-02-10T01:24:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-02-10T01:24:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:24:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:24:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:24:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-02-10T01:24:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-02-10T01:24:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:24:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:24:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:24:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-02-10T01:24:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-02-10T01:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_10.status 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_10.status 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.status 2024-02-10T01:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_10.status 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_10.status 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-02-10T01:24:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 105 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.status 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.status 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-02-10T01:24:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=105 2024-02-10T01:24:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-02-10T01:24:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=45 2024-02-10T01:24:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 105 = 45 2024-02-10T01:24:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-02-10T01:24:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 105 = '' 2024-02-10T01:24:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-02-10T01:24:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 105 = '' 2024-02-10T01:24:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:37.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2024-02-10T01:24:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.status 2024-02-10T01:24:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-02-10T01:24:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=105 2024-02-10T01:24:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 105 = 105 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-02-10T01:24:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_10': 2024-02-10T01:24:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165126a3d7 2024-02-10T01:24:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165126a3d7 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:17 2024 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:17 2024 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:17 2024 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:24:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_10 snap 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_10 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.export 2024-02-10T01:24:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.export 2024-02-10T01:24:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.export 2024-02-10T01:24:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.export 2024-02-10T01:24:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.export 2024-02-10T01:24:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.export 2024-02-10T01:24:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_10.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_10.export 2024-02-10T01:24:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_11 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_11 snap 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-02-10T01:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-02-10T01:24:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-02-10T01:24:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-02-10T01:24:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-02-10T01:24:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-02-10T01:24:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-02-10T01:24:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:24:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:24:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:24:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:24:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:24:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:24:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-02-10T01:24:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-02-10T01:24:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_11.status 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_11.status 2024-02-10T01:24:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.status 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_11.status 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_11.status 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-02-10T01:24:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 107 2024-02-10T01:24:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.status 2024-02-10T01:24:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.status 2024-02-10T01:24:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-02-10T01:24:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=107 2024-02-10T01:24:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-02-10T01:24:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=46 2024-02-10T01:24:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 107 = 46 2024-02-10T01:24:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:43.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2024-02-10T01:24:43.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9399068349337143) > weight (0.25) 2024-02-10T01:24:43.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5354086672386562) > weight (0.25) 2024-02-10T01:24:43.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7545610471041149) > weight (0.25) 2024-02-10T01:24:43.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.44176501701463133) > weight (0.25) 2024-02-10T01:24:43.539 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-02-10T01:24:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-02-10T01:24:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=46 2024-02-10T01:24:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 107 = 46 2024-02-10T01:24:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-02-10T01:24:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 107 = '' 2024-02-10T01:24:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.status 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-02-10T01:24:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=107 2024-02-10T01:24:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 107 = 107 2024-02-10T01:24:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-02-10T01:24:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_11': 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa4d6b3ff3 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa4d6b3ff3 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:26 2024 2024-02-10T01:24:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:26 2024 2024-02-10T01:24:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:26 2024 2024-02-10T01:24:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:24:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_11 snap 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_11 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.export 2024-02-10T01:24:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.export 2024-02-10T01:24:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.export 2024-02-10T01:24:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.export 2024-02-10T01:24:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.export 2024-02-10T01:24:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.export 2024-02-10T01:24:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_11.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_11.export 2024-02-10T01:24:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_12 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_12 snap 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-02-10T01:24:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:48.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2024-02-10T01:24:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-02-10T01:24:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-02-10T01:24:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-02-10T01:24:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-02-10T01:24:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-02-10T01:24:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-02-10T01:24:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:24:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:24:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-02-10T01:24:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-02-10T01:24:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:24:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:24:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:24:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2024-02-10T01:24:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-02-10T01:24:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-02-10T01:24:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_12.status 2024-02-10T01:24:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_12.status 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.status 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_12.status 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_12.status 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-02-10T01:24:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 109 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.status 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.status 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-02-10T01:24:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=109 2024-02-10T01:24:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-02-10T01:24:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=47 2024-02-10T01:24:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 109 = 47 2024-02-10T01:24:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-02-10T01:24:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:50.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3779685122934193) > weight (0.25) 2024-02-10T01:24:50.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2623309383145477) > weight (0.25) 2024-02-10T01:24:50.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.980513693343581) > weight (0.25) 2024-02-10T01:24:50.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7992056123651212) > weight (0.25) 2024-02-10T01:24:50.724 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-02-10T01:24:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=47 2024-02-10T01:24:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 109 = 47 2024-02-10T01:24:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-02-10T01:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 109 = '' 2024-02-10T01:24:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.status 2024-02-10T01:24:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-02-10T01:24:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=109 2024-02-10T01:24:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 109 = 109 2024-02-10T01:24:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:24:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:54.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2024-02-10T01:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-02-10T01:24:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_12': 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201c430258b 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201c430258b 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:39 2024 2024-02-10T01:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:39 2024 2024-02-10T01:24:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:39 2024 2024-02-10T01:24:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:24:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:24:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:24:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:24:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_12 snap 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_12 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.export 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.export 2024-02-10T01:24:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.export 2024-02-10T01:24:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.export 2024-02-10T01:24:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.export 2024-02-10T01:24:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:24:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.export 2024-02-10T01:24:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_12.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_12.export 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_13 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_13 snap 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:24:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:24:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:24:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:24:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:24:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:24:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:24:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:24:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-02-10T01:24:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-02-10T01:24:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:24:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:24:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:24:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:24:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-02-10T01:24:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-02-10T01:24:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:24:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:24:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:24:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:24:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:24:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:24:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:24:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:24:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-02-10T01:24:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:24:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:24:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:24:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:24:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:24:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:24:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:24:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_13.status 2024-02-10T01:24:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_13.status 2024-02-10T01:24:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.status 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_13.status 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_13.status 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-02-10T01:24:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 111 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.status 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.status 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-02-10T01:24:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:24:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=111 2024-02-10T01:24:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-02-10T01:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=48 2024-02-10T01:24:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 111 = 48 2024-02-10T01:24:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:24:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-02-10T01:24:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 111 = '' 2024-02-10T01:24:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:24:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-02-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:24:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:24:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:24:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:24:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:24:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:24:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:24:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 111 = '' 2024-02-10T01:24:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:24:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:24:59.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2024-02-10T01:25:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:25:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:25:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.status 2024-02-10T01:25:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-02-10T01:25:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=111 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 111 = 111 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:25:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_13': 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 128631c941d5 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.128631c941d5 2024-02-10T01:25:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:53 2024 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:15:53 2024 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:53 2024 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:25:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_13 snap 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_13 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.export 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.export 2024-02-10T01:25:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.export 2024-02-10T01:25:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.export 2024-02-10T01:25:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.export 2024-02-10T01:25:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.export 2024-02-10T01:25:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_13.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_13.export 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_14 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_14 snap 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-02-10T01:25:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:04.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-02-10T01:25:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-02-10T01:25:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-02-10T01:25:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:25:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:25:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:25:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-02-10T01:25:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-02-10T01:25:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:25:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:25:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:25:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-02-10T01:25:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-02-10T01:25:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:25:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:25:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:25:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:25:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:25:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:25:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:25:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:25:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:25:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-02-10T01:25:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_14.status 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_14.status 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.status 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_14.status 2024-02-10T01:25:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_14.status 2024-02-10T01:25:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-02-10T01:25:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-02-10T01:25:04.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2024-02-10T01:25:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 113 2024-02-10T01:25:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.status 2024-02-10T01:25:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.status 2024-02-10T01:25:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-02-10T01:25:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=113 2024-02-10T01:25:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-02-10T01:25:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=49 2024-02-10T01:25:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 113 = 49 2024-02-10T01:25:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-02-10T01:25:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 113 = '' 2024-02-10T01:25:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-02-10T01:25:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 113 = '' 2024-02-10T01:25:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:25:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.status 2024-02-10T01:25:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-02-10T01:25:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=113 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 113 = 113 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_14': 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1286a0cf4358 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1286a0cf4358 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:05 2024 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:05 2024 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:05 2024 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:25:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_14 snap 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_14 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.export 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.export 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.export 2024-02-10T01:25:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.export 2024-02-10T01:25:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.export 2024-02-10T01:25:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.export 2024-02-10T01:25:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_14.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_14.export 2024-02-10T01:25:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_15 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_15 snap 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:10.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2024-02-10T01:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-02-10T01:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-02-10T01:25:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-02-10T01:25:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-02-10T01:25:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-02-10T01:25:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:25:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:25:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:25:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:25:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-02-10T01:25:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_15.status 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_15.status 2024-02-10T01:25:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.status 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_15.status 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_15.status 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-02-10T01:25:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 115 2024-02-10T01:25:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.status 2024-02-10T01:25:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.status 2024-02-10T01:25:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-02-10T01:25:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=115 2024-02-10T01:25:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-02-10T01:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=50 2024-02-10T01:25:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 115 = 50 2024-02-10T01:25:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-02-10T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=50 2024-02-10T01:25:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 115 = 50 2024-02-10T01:25:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-02-10T01:25:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 115 = '' 2024-02-10T01:25:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:25:15.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2024-02-10T01:25:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.status 2024-02-10T01:25:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-02-10T01:25:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=115 2024-02-10T01:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 115 = 115 2024-02-10T01:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-02-10T01:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_15': 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201eb3deb91 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201eb3deb91 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:18 2024 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:18 2024 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:18 2024 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:25:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_15 snap 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_15 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.export 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.export 2024-02-10T01:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.export 2024-02-10T01:25:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.export 2024-02-10T01:25:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.export 2024-02-10T01:25:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.export 2024-02-10T01:25:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_15.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_15.export 2024-02-10T01:25:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_16 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_16 snap 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-02-10T01:25:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-02-10T01:25:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-02-10T01:25:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-02-10T01:25:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-02-10T01:25:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_16.status 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_16.status 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.status 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_16.status 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_16.status 2024-02-10T01:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-02-10T01:25:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-02-10T01:25:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 117 2024-02-10T01:25:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.status 2024-02-10T01:25:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.status 2024-02-10T01:25:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-02-10T01:25:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=117 2024-02-10T01:25:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:21.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2024-02-10T01:25:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-02-10T01:25:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=51 2024-02-10T01:25:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 117 = 51 2024-02-10T01:25:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-02-10T01:25:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=51 2024-02-10T01:25:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 117 = 51 2024-02-10T01:25:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-02-10T01:25:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 117 = '' 2024-02-10T01:25:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:25:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.status 2024-02-10T01:25:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-02-10T01:25:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=117 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 117 = 117 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_16': 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12863cb0a88b 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12863cb0a88b 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:30 2024 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:30 2024 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:30 2024 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:25:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_16 snap 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_16 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.export 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.export 2024-02-10T01:25:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.export 2024-02-10T01:25:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.export 2024-02-10T01:25:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.export 2024-02-10T01:25:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.export 2024-02-10T01:25:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_16.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_16.export 2024-02-10T01:25:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_17 2024-02-10T01:25:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_17 snap 2024-02-10T01:25:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:26.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-02-10T01:25:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-02-10T01:25:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-02-10T01:25:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:25:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:25:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:25:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-02-10T01:25:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-02-10T01:25:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-02-10T01:25:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:25:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:25:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:25:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:25:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-02-10T01:25:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-02-10T01:25:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:25:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:25:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:25:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:25:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:25:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:25:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:25:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2024-02-10T01:25:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-02-10T01:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_17.status 2024-02-10T01:25:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_17.status 2024-02-10T01:25:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.status 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_17.status 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_17.status 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-02-10T01:25:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-02-10T01:25:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 119 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.status 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.status 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-02-10T01:25:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=119 2024-02-10T01:25:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-02-10T01:25:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=52 2024-02-10T01:25:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 119 = 52 2024-02-10T01:25:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-02-10T01:25:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 119 = '' 2024-02-10T01:25:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-02-10T01:25:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 119 = '' 2024-02-10T01:25:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:25:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.status 2024-02-10T01:25:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-02-10T01:25:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=119 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 119 = 119 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:32.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2024-02-10T01:25:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_17': 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165f6384a7a 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165f6384a7a 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:39 2024 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:39 2024 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:39 2024 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:25:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_17 snap 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_17 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.export 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.export 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.export 2024-02-10T01:25:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.export 2024-02-10T01:25:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.export 2024-02-10T01:25:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.export 2024-02-10T01:25:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_17.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_17.export 2024-02-10T01:25:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_18 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_18 snap 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-02-10T01:25:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-02-10T01:25:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:25:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-02-10T01:25:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-02-10T01:25:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-02-10T01:25:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:25:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:25:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:25:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:25:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:25:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:25:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:25:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:25:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:25:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:25:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:25:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:25:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:25:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:25:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:25:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:25:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:25:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-02-10T01:25:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_18.status 2024-02-10T01:25:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_18.status 2024-02-10T01:25:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.status 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_18.status 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_18.status 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-02-10T01:25:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 121 2024-02-10T01:25:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.status 2024-02-10T01:25:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.status 2024-02-10T01:25:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-02-10T01:25:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=121 2024-02-10T01:25:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-02-10T01:25:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 121 = '' 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-02-10T01:25:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 121 = '' 2024-02-10T01:25:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:37.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2024-02-10T01:25:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.status 2024-02-10T01:25:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-02-10T01:25:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=121 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 121 = 121 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-02-10T01:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_18': 2024-02-10T01:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aafc5876ef 2024-02-10T01:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aafc5876ef 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:48 2024 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:48 2024 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:48 2024 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:25:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_18 snap 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_18 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.export 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.export 2024-02-10T01:25:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.export 2024-02-10T01:25:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.export 2024-02-10T01:25:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.export 2024-02-10T01:25:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.export 2024-02-10T01:25:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_18.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_18.export 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_19 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_19 snap 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-02-10T01:25:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-02-10T01:25:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:25:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:25:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:25:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:25:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:25:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:25:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:25:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-02-10T01:25:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_19.status 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_19.status 2024-02-10T01:25:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.status 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_19.status 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_19.status 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-02-10T01:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 123 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.status 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.status 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-02-10T01:25:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=123 2024-02-10T01:25:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-02-10T01:25:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 123 = '' 2024-02-10T01:25:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-02-10T01:25:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 123 = '' 2024-02-10T01:25:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:42.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.status 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-02-10T01:25:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=123 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 123 = 123 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-02-10T01:25:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_19': 2024-02-10T01:25:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201a7e8981b 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201a7e8981b 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:55 2024 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:16:55 2024 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:55 2024 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:25:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_19 snap 2024-02-10T01:25:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_19 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.export 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.export 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.export 2024-02-10T01:25:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.export 2024-02-10T01:25:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.export 2024-02-10T01:25:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.export 2024-02-10T01:25:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_19.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_19.export 2024-02-10T01:25:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_20 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_20 snap 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-02-10T01:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-02-10T01:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-02-10T01:25:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_20.status 2024-02-10T01:25:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_20.status 2024-02-10T01:25:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.status 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_20.status 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_20.status 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-02-10T01:25:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 125 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.status 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.status 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-02-10T01:25:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=125 2024-02-10T01:25:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-02-10T01:25:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=55 2024-02-10T01:25:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 125 = 55 2024-02-10T01:25:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-02-10T01:25:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 125 = '' 2024-02-10T01:25:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:48.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2024-02-10T01:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-02-10T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 125 = '' 2024-02-10T01:25:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:25:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.status 2024-02-10T01:25:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-02-10T01:25:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=125 2024-02-10T01:25:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 125 = 125 2024-02-10T01:25:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-02-10T01:25:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-02-10T01:25:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_20': 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12863cb65f06 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12863cb65f06 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:03 2024 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:03 2024 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:03 2024 2024-02-10T01:25:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:25:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_20 snap 2024-02-10T01:25:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_20 2024-02-10T01:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.export 2024-02-10T01:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.export 2024-02-10T01:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.export 2024-02-10T01:25:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.export 2024-02-10T01:25:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.export 2024-02-10T01:25:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.export 2024-02-10T01:25:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_20.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_20.export 2024-02-10T01:25:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_21 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_21 snap 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-02-10T01:25:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-02-10T01:25:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:53.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:25:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:25:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:25:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:25:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:25:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:25:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-02-10T01:25:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:25:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:25:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:25:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:25:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:25:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:25:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:25:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_21.status 2024-02-10T01:25:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_21.status 2024-02-10T01:25:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.status 2024-02-10T01:25:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_21.status 2024-02-10T01:25:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_21.status 2024-02-10T01:25:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:25:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-02-10T01:25:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-02-10T01:25:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 127 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.status 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.status 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-02-10T01:25:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:25:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=127 2024-02-10T01:25:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:25:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-02-10T01:25:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 127 = '' 2024-02-10T01:25:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:25:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-02-10T01:25:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:25:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 127 = '' 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:25:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:25:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.status 2024-02-10T01:25:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-02-10T01:25:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:25:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:25:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=127 2024-02-10T01:25:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 127 = 127 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:25:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-02-10T01:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_21': 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165d7c7ae39 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165d7c7ae39 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:14 2024 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:14 2024 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:14 2024 2024-02-10T01:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:25:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:25:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:25:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:25:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:25:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_21 snap 2024-02-10T01:25:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_21 2024-02-10T01:25:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:25:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.export 2024-02-10T01:25:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.export 2024-02-10T01:25:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.export 2024-02-10T01:25:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.export 2024-02-10T01:25:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.export 2024-02-10T01:25:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:25:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.export 2024-02-10T01:25:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_21.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_21.export 2024-02-10T01:25:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:25:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:25:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_22 2024-02-10T01:25:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_22 snap 2024-02-10T01:25:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:25:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:25:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:25:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:25:58.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7588942806480345) > weight (0.25) 2024-02-10T01:25:58.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3753939740359925) > weight (0.25) 2024-02-10T01:25:58.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8946822947718136) > weight (0.25) 2024-02-10T01:25:58.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4458248628513545) > weight (0.25) 2024-02-10T01:25:58.541 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-02-10T01:25:59.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:25:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:25:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:25:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:25:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:25:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-02-10T01:26:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-02-10T01:26:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-02-10T01:26:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-02-10T01:26:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_22.status 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_22.status 2024-02-10T01:26:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.status 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_22.status 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_22.status 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-02-10T01:26:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 129 2024-02-10T01:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.status 2024-02-10T01:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.status 2024-02-10T01:26:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-02-10T01:26:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=129 2024-02-10T01:26:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-02-10T01:26:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=57 2024-02-10T01:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 129 = 57 2024-02-10T01:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:01.724 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-02-10T01:26:01.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-02-10T01:26:01.725 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-02-10T01:26:01.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:2024-02-10T01:26:01.725+0000 7f28a1c06640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 49109) UID: 1000 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-02-10T01:26:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 129 = '' 2024-02-10T01:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-02-10T01:26:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 129 = '' 2024-02-10T01:26:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:26:04.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.status 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-02-10T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=129 2024-02-10T01:26:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 129 = 129 2024-02-10T01:26:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-02-10T01:26:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_22': 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aafdc11b2c 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aafdc11b2c 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:28 2024 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:28 2024 2024-02-10T01:26:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:28 2024 2024-02-10T01:26:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:26:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_22 snap 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_22 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.export 2024-02-10T01:26:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.export 2024-02-10T01:26:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.export 2024-02-10T01:26:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.export 2024-02-10T01:26:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.export 2024-02-10T01:26:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.export 2024-02-10T01:26:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_22.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_22.export 2024-02-10T01:26:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_23 2024-02-10T01:26:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_23 snap 2024-02-10T01:26:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-02-10T01:26:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-02-10T01:26:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-02-10T01:26:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-02-10T01:26:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_23.status 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_23.status 2024-02-10T01:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.status 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_23.status 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_23.status 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-02-10T01:26:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 131 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.status 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.status 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-02-10T01:26:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=131 2024-02-10T01:26:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-02-10T01:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=58 2024-02-10T01:26:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 131 = 58 2024-02-10T01:26:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:08.725 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-02-10T01:26:08.726 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:26:08.726 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-02-10T01:26:08.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-02-10T01:26:08.727 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-02-10T01:26:08.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-02-10T01:26:08.729 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-02-10T01:26:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-02-10T01:26:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 131 = '' 2024-02-10T01:26:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:09.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-02-10T01:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 131 = '' 2024-02-10T01:26:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.status 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-02-10T01:26:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=131 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 131 = 131 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_23': 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201676a5a41 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201676a5a41 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:38 2024 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:38 2024 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:38 2024 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:26:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_23 snap 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_23 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.export 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.export 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.export 2024-02-10T01:26:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.export 2024-02-10T01:26:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.export 2024-02-10T01:26:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.export 2024-02-10T01:26:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_23.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_23.export 2024-02-10T01:26:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_24 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_24 snap 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-02-10T01:26:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-02-10T01:26:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_24.status 2024-02-10T01:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_24.status 2024-02-10T01:26:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.status 2024-02-10T01:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_24.status 2024-02-10T01:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_24.status 2024-02-10T01:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-02-10T01:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-02-10T01:26:15.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2024-02-10T01:26:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 133 2024-02-10T01:26:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.status 2024-02-10T01:26:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.status 2024-02-10T01:26:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-02-10T01:26:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=133 2024-02-10T01:26:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-02-10T01:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 133 = '' 2024-02-10T01:26:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-02-10T01:26:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 133 = '' 2024-02-10T01:26:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.status 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-02-10T01:26:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=133 2024-02-10T01:26:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 133 = 133 2024-02-10T01:26:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-02-10T01:26:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_24': 2024-02-10T01:26:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1286c08399a4 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1286c08399a4 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:51 2024 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:17:51 2024 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:51 2024 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:26:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_24 snap 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_24 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.export 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.export 2024-02-10T01:26:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.export 2024-02-10T01:26:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.export 2024-02-10T01:26:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.export 2024-02-10T01:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.export 2024-02-10T01:26:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_24.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_24.export 2024-02-10T01:26:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_25 2024-02-10T01:26:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_25 snap 2024-02-10T01:26:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-02-10T01:26:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-02-10T01:26:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:20.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2024-02-10T01:26:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-02-10T01:26:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-02-10T01:26:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_25.status 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_25.status 2024-02-10T01:26:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.status 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_25.status 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_25.status 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-02-10T01:26:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 135 2024-02-10T01:26:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.status 2024-02-10T01:26:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.status 2024-02-10T01:26:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-02-10T01:26:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=135 2024-02-10T01:26:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-02-10T01:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 135 = '' 2024-02-10T01:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-02-10T01:26:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 135 = '' 2024-02-10T01:26:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.status 2024-02-10T01:26:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-02-10T01:26:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=135 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 135 = 135 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-02-10T01:26:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_25': 2024-02-10T01:26:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165fd99f663 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165fd99f663 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:01 2024 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:01 2024 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:01 2024 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:26:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_25 snap 2024-02-10T01:26:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_25 2024-02-10T01:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.export 2024-02-10T01:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.export 2024-02-10T01:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.export 2024-02-10T01:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.export 2024-02-10T01:26:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.export 2024-02-10T01:26:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.export 2024-02-10T01:26:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_25.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_25.export 2024-02-10T01:26:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_26 2024-02-10T01:26:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_26 snap 2024-02-10T01:26:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-02-10T01:26:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-02-10T01:26:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:26.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2024-02-10T01:26:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-02-10T01:26:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-02-10T01:26:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-02-10T01:26:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-02-10T01:26:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-02-10T01:26:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_26.status 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_26.status 2024-02-10T01:26:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.status 2024-02-10T01:26:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_26.status 2024-02-10T01:26:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_26.status 2024-02-10T01:26:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-02-10T01:26:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-02-10T01:26:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 137 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.status 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.status 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-02-10T01:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=137 2024-02-10T01:26:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-02-10T01:26:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 137 = '' 2024-02-10T01:26:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-02-10T01:26:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 137 = '' 2024-02-10T01:26:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.status 2024-02-10T01:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-02-10T01:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=137 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 137 = 137 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-02-10T01:26:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_26': 2024-02-10T01:26:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa684a031a 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa684a031a 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:09 2024 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:09 2024 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:09 2024 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:26:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_26 snap 2024-02-10T01:26:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_26 2024-02-10T01:26:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.export 2024-02-10T01:26:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.export 2024-02-10T01:26:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.export 2024-02-10T01:26:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.export 2024-02-10T01:26:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.export 2024-02-10T01:26:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.export 2024-02-10T01:26:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_26.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_26.export 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_27 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_27 snap 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2024-02-10T01:26:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-02-10T01:26:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-02-10T01:26:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-02-10T01:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-02-10T01:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-02-10T01:26:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_27.status 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_27.status 2024-02-10T01:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.status 2024-02-10T01:26:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_27.status 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_27.status 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-02-10T01:26:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 139 2024-02-10T01:26:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.status 2024-02-10T01:26:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.status 2024-02-10T01:26:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-02-10T01:26:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=139 2024-02-10T01:26:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-02-10T01:26:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 139 = '' 2024-02-10T01:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-02-10T01:26:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 139 = '' 2024-02-10T01:26:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.status 2024-02-10T01:26:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-02-10T01:26:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=139 2024-02-10T01:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 139 = 139 2024-02-10T01:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-02-10T01:26:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_27': 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295f903f63f 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295f903f63f 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:45 2024 2024-02-10T01:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:45 2024 2024-02-10T01:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:45 2024 2024-02-10T01:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_27 snap 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_27 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.export 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.export 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.export 2024-02-10T01:26:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.export 2024-02-10T01:26:36.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2024-02-10T01:26:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.export 2024-02-10T01:26:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.export 2024-02-10T01:26:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_27.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_27.export 2024-02-10T01:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_28 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_28 snap 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-02-10T01:26:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-02-10T01:26:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-02-10T01:26:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-02-10T01:26:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-02-10T01:26:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-02-10T01:26:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_28.status 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_28.status 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.status 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_28.status 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_28.status 2024-02-10T01:26:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-02-10T01:26:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-02-10T01:26:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 141 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.status 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.status 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-02-10T01:26:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=141 2024-02-10T01:26:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-02-10T01:26:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=63 2024-02-10T01:26:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 141 = 63 2024-02-10T01:26:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-02-10T01:26:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=63 2024-02-10T01:26:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 141 = 63 2024-02-10T01:26:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-02-10T01:26:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 141 = '' 2024-02-10T01:26:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:26:42.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.status 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-02-10T01:26:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=141 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 141 = 141 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_28': 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295c4ad4b3e 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295c4ad4b3e 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:34 2024 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:34 2024 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:34 2024 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_28 snap 2024-02-10T01:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_28 2024-02-10T01:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.export 2024-02-10T01:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.export 2024-02-10T01:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.export 2024-02-10T01:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.export 2024-02-10T01:26:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.export 2024-02-10T01:26:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.export 2024-02-10T01:26:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_28.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_28.export 2024-02-10T01:26:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_29 2024-02-10T01:26:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_29 snap 2024-02-10T01:26:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-02-10T01:26:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-02-10T01:26:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-02-10T01:26:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-02-10T01:26:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-02-10T01:26:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-02-10T01:26:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:26:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:26:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:26:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:26:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:26:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:26:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:26:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-02-10T01:26:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_29.status 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_29.status 2024-02-10T01:26:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.status 2024-02-10T01:26:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_29.status 2024-02-10T01:26:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_29.status 2024-02-10T01:26:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-02-10T01:26:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-02-10T01:26:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 143 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.status 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.status 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-02-10T01:26:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=143 2024-02-10T01:26:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-02-10T01:26:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=64 2024-02-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 143 = 64 2024-02-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:47.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-02-10T01:26:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 143 = '' 2024-02-10T01:26:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-02-10T01:26:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 143 = '' 2024-02-10T01:26:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:26:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.status 2024-02-10T01:26:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-02-10T01:26:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=143 2024-02-10T01:26:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 143 = 143 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_29': 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 116597bb13ad 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.116597bb13ad 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:46 2024 2024-02-10T01:26:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:18:46 2024 2024-02-10T01:26:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:46 2024 2024-02-10T01:26:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:26:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_29 snap 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_29 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.export 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.export 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.export 2024-02-10T01:26:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.export 2024-02-10T01:26:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.export 2024-02-10T01:26:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.export 2024-02-10T01:26:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_29.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_29.export 2024-02-10T01:26:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:26:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_30 2024-02-10T01:26:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_30 snap 2024-02-10T01:26:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:26:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-02-10T01:26:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:53.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:26:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:26:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:26:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-02-10T01:26:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:26:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:26:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:26:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:26:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:26:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-02-10T01:26:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-02-10T01:26:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-02-10T01:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:26:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:26:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:26:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-02-10T01:26:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-02-10T01:26:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:26:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:26:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:26:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-02-10T01:26:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:26:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:26:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:26:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:26:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:26:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:26:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:26:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:26:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:26:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-02-10T01:26:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:26:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:26:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:26:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:26:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-02-10T01:26:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:26:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_30.status 2024-02-10T01:26:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_30.status 2024-02-10T01:26:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.status 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_30.status 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_30.status 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-02-10T01:26:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 145 2024-02-10T01:26:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.status 2024-02-10T01:26:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:26:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.status 2024-02-10T01:26:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-02-10T01:26:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:26:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=145 2024-02-10T01:26:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:26:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-02-10T01:26:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=65 2024-02-10T01:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 145 = 65 2024-02-10T01:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-02-10T01:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 145 = '' 2024-02-10T01:26:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-02-10T01:26:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:26:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:26:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:26:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:26:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:26:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 145 = '' 2024-02-10T01:26:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:26:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:26:58.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.status 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-02-10T01:26:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:26:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:26:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=145 2024-02-10T01:26:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 145 = 145 2024-02-10T01:26:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:26:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:26:59.541 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:26:59.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:26:59.542 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-02-10T01:26:59.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:26:59.542+0000 7ff428493640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 53189) UID: 1000 2024-02-10T01:26:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-02-10T01:26:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_30': 2024-02-10T01:26:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa8142e435 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa8142e435 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:01 2024 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:01 2024 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:01 2024 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:26:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:26:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:26:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:26:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_30 snap 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_30 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.export 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.export 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.export 2024-02-10T01:26:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.export 2024-02-10T01:26:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:26:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.export 2024-02-10T01:27:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.export 2024-02-10T01:27:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_30.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_30.export 2024-02-10T01:27:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_31 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_31 snap 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-02-10T01:27:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:27:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:27:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:27:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-02-10T01:27:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:27:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:27:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:27:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:27:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:27:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:27:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-02-10T01:27:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-02-10T01:27:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-02-10T01:27:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:27:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:27:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:27:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:27:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-02-10T01:27:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-02-10T01:27:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:27:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:27:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:27:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:27:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:27:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:27:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:27:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-02-10T01:27:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-02-10T01:27:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:27:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:27:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2024-02-10T01:27:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T01:27:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:3 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:3 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 3 = cluster1:3 ']' 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=3 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=3 2024-02-10T01:27:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-02-10T01:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-02-10T01:27:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2024-02-10T01:27:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:27:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:27:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:27:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:27:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:27:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:27:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_31.status 2024-02-10T01:27:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_31.status 2024-02-10T01:27:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.status 2024-02-10T01:27:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_31.status 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_31.status 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-02-10T01:27:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 147 2024-02-10T01:27:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.status 2024-02-10T01:27:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.status 2024-02-10T01:27:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-02-10T01:27:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:27:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=147 2024-02-10T01:27:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:27:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-02-10T01:27:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=66 2024-02-10T01:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 147 = 66 2024-02-10T01:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:27:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-02-10T01:27:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=66 2024-02-10T01:27:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 147 = 66 2024-02-10T01:27:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:27:03.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2024-02-10T01:27:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-02-10T01:27:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 147 = '' 2024-02-10T01:27:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:27:05.543 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:27:05.543 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:27:05.543 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-02-10T01:27:05.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-02-10T01:27:05.544 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T01:27:05.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-02-10T01:27:05.547 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-02-10T01:27:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.status 2024-02-10T01:27:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-02-10T01:27:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=147 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 147 = 147 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:27:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-02-10T01:27:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_31': 2024-02-10T01:27:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:27:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:27:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:27:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12863cc0137b 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12863cc0137b 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:13 2024 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:13 2024 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:13 2024 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:27:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:27:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_31 snap 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_31 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.export 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.export 2024-02-10T01:27:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.export 2024-02-10T01:27:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.export 2024-02-10T01:27:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.export 2024-02-10T01:27:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.export 2024-02-10T01:27:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_31.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_31.export 2024-02-10T01:27:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_32 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_32 snap 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:27:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-02-10T01:27:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-02-10T01:27:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:27:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:27:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:27:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:27:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:27:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:27:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:27:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:27:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-02-10T01:27:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-02-10T01:27:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:27:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:27:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-02-10T01:27:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:27:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:27:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:27:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:27:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:27:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-02-10T01:27:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:27:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:27:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:27:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:27:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:27:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_32.status 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_32.status 2024-02-10T01:27:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.status 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_32.status 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_32.status 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-02-10T01:27:09.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2024-02-10T01:27:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 149 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.status 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.status 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-02-10T01:27:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=149 2024-02-10T01:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:27:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-02-10T01:27:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 149 = '' 2024-02-10T01:27:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:27:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-02-10T01:27:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 149 = '' 2024-02-10T01:27:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:27:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.status 2024-02-10T01:27:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-02-10T01:27:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=149 2024-02-10T01:27:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 149 = 149 2024-02-10T01:27:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_32': 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12956af0d92f 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12956af0d92f 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:27:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:27 2024 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:27 2024 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:27 2024 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:27:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_32 snap 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_32 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.export 2024-02-10T01:27:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.export 2024-02-10T01:27:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.export 2024-02-10T01:27:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.export 2024-02-10T01:27:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.export 2024-02-10T01:27:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.export 2024-02-10T01:27:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_32.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_32.export 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_33 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_33 snap 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:27:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:14.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2024-02-10T01:27:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-02-10T01:27:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:27:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:27:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:27:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:27:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:27:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:27:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:27:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:27:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-02-10T01:27:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-02-10T01:27:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:27:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:27:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 1 = cluster1:1 ']' 2024-02-10T01:27:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=1 2024-02-10T01:27:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=1 2024-02-10T01:27:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-02-10T01:27:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-02-10T01:27:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-02-10T01:27:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-02-10T01:27:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd mirror status : get status for rbd mirror 2024-02-10T01:27:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:admin_socket: invalid command 2024-02-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-02-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:2 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:2 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 2 = cluster1:2 ']' 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=2 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=2 2024-02-10T01:27:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-02-10T01:27:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:27:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:27:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-02-10T01:27:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-02-10T01:27:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:27:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:27:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:27:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:27:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_33.status 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_33.status 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.status 2024-02-10T01:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_33.status 2024-02-10T01:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_33.status 2024-02-10T01:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-02-10T01:27:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-02-10T01:27:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 151 2024-02-10T01:27:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.status 2024-02-10T01:27:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.status 2024-02-10T01:27:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-02-10T01:27:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=151 2024-02-10T01:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-02-10T01:27:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=68 2024-02-10T01:27:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 151 = 68 2024-02-10T01:27:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:27:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-02-10T01:27:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 151 = '' 2024-02-10T01:27:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-02-10T01:27:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 151 = '' 2024-02-10T01:27:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:27:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.status 2024-02-10T01:27:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-02-10T01:27:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=151 2024-02-10T01:27:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 151 = 151 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:27:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:20.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2024-02-10T01:27:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-02-10T01:27:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_33': 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 116556c1ef13 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.116556c1ef13 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:27:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:27:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:27:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:27:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:34 2024 2024-02-10T01:27:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:34 2024 2024-02-10T01:27:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:34 2024 2024-02-10T01:27:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:27:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:27:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:27:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:27:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:27:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_33 snap 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_33 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.export 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.export 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.export 2024-02-10T01:27:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.export 2024-02-10T01:27:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.export 2024-02-10T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-02-10T01:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cmp /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.export 2024-02-10T01:27:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_33.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_33.export 2024-02-10T01:27:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ image=image_34 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_snap cluster2 mirror image_34 snap 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-02-10T01:27:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-02-10T01:27:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state=started 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-02-10T01:27:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local test_state=started 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_result 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local current_state=stopped 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_instance=cluster1 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance=cluster1 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local loop_instance 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ grep -i state 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-02-10T01:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sleep 0 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ seq 0 3 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ shift 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T01:27:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local local_cluster=cluster1:0 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-02-10T01:27:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=local_cluster 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval local_cluster=cluster1 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ local_cluster=cluster1 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-02-10T01:27:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-02-10T01:27:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-02-10T01:27:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-02-10T01:27:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ return 0 2024-02-10T01:27:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status_result=' "state": "Replaying"' 2024-02-10T01:27:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo ' "state": "Replaying"' 2024-02-10T01:27:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -i Replaying 2024-02-10T01:27:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "state": "Replaying" 2024-02-10T01:27:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ current_state=started 2024-02-10T01:27:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test started = started 2024-02-10T01:27:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = snapshot ']' 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_cluster=cluster1 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster2-mirror-image_34.status 2024-02-10T01:27:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster2-mirror-image_34.status 2024-02-10T01:27:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status_log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_34.status 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mkfname cluster1-mirror-image_34.status 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo cluster1-mirror-image_34.status 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -e 's|[/ ]|_|g' 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local local_status_log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-02-10T01:27:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 153 2024-02-10T01:27:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_34.status 2024-02-10T01:27:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T01:27:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_34.status 2024-02-10T01:27:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-02-10T01:27:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-02-10T01:27:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snapshot_id=153 2024-02-10T01:27:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.2 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-02-10T01:27:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=69 2024-02-10T01:27:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 153 = 69 2024-02-10T01:27:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.4 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-02-10T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 153 = '' 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 0.8 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-02-10T01:27:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ true 2024-02-10T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:-:1.1: Document is empty 2024-02-10T01:27:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-02-10T01:27:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:^ 2024-02-10T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id= 2024-02-10T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 153 = '' 2024-02-10T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-02-10T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1.6 2024-02-10T01:27:25.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2024-02-10T01:27:25.503 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-02-10T01:27:25.907 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2024-02-10T01:27:25.908 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2024-02-10T01:27:25.908 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2024-02-10T01:27:25.908 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2024-02-10T01:27:25.909 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:27:25.907+0000 7f1f5391e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 34084) UID: 0 2024-02-10T01:27:25.909 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:27:25.907+0000 7f1f5391e640 -1 osd.1 222 *** Got signal Terminated *** 2024-02-10T01:27:25.909 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-02-10T01:27:25.907+0000 7f1f5391e640 -1 osd.1 222 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T01:27:25.909 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:27:25.908+0000 7fe72d2fa640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 34086) UID: 0 2024-02-10T01:27:25.909 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:27:25.908+0000 7fe72d2fa640 -1 osd.2 222 *** Got signal Terminated *** 2024-02-10T01:27:25.909 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-02-10T01:27:25.908+0000 7fe72d2fa640 -1 osd.2 222 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T01:27:25.910 INFO:tasks.ceph.cluster2.mon.a.smithi060.stderr:2024-02-10T01:27:25.908+0000 7f0b2ee6d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 33776) UID: 0 2024-02-10T01:27:25.910 INFO:tasks.ceph.cluster2.mon.a.smithi060.stderr:2024-02-10T01:27:25.908+0000 7f0b2ee6d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T01:27:25.910 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:27:25.909+0000 7f50f6b09640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 38033) UID: 0 2024-02-10T01:27:25.910 INFO:tasks.ceph.cluster2.mgr.x.smithi008.stderr:2024-02-10T01:27:25.909+0000 7f50f6b09640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local log=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.status 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-02-10T01:27:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-02-10T01:27:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local primary_snapshot_id=153 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 153 = 153 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local s 2024-02-10T01:27:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-02-10T01:27:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-02-10T01:27:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-02-10T01:27:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_34': 2024-02-10T01:27:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:27:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:27:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:27:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 129f93ae6656 2024-02-10T01:27:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.129f93ae6656 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:49 2024 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:49 2024 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:49 2024 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: protected: False 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:27:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_image_snaps mirror image_34 snap 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image_34 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local snap_name=snap 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local rmt_export=/tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_34.export 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local loc_export=/tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.export 2024-02-10T01:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_34.export /tmp/tmp.ywJfcqGoUP/cluster1-mirror-image_34.export 2024-02-10T01:27:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.ywJfcqGoUP/cluster2-mirror-image_34.export 2024-02-10T01:27:39.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8151920799409494) > weight (0.25) 2024-02-10T01:27:39.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.801171338952714) > weight (0.25) 2024-02-10T01:27:39.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6445990934103585) > weight (0.25) 2024-02-10T01:27:39.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8617672123361771) > weight (0.25) 2024-02-10T01:27:39.730 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-02-10T01:28:42.731 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-02-10T01:28:42.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-02-10T01:28:42.733 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-02-10T01:28:42.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:2024-02-10T01:28:42.733+0000 7fb47bb02640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57934) UID: 1000 2024-02-10T01:28:47.734 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-02-10T01:28:47.734 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:28:58.547 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-02-10T01:28:58.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-02-10T01:28:58.548 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-02-10T01:28:58.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:28:58.548+0000 7faf51066640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59960) UID: 1000 2024-02-10T01:29:02.549 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-02-10T01:29:02.549 INFO:teuthology.orchestra.run:waiting for 600 2024-02-10T01:32:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cleanup 110 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code=110 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 110 -ne 0 ']' 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster daemon image_pool image_ns image 2024-02-10T01:32:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-02-10T01:32:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 status' 2024-02-10T01:32:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 status 2024-02-10T01:32:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 -s 2024-02-10T01:32:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: cluster: 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: c022c2cc-868d-45e0-bbf6-c3f8e32882f5 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: HEALTH_OK 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: services: 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mon: 1 daemons, quorum a (age 31m) 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mgr: x(active, since 31m) 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: osd: 3 osds: 3 up (since 31m), 3 in (since 31m) 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: rbd-mirror: 4 daemons active (1 hosts) 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: data: 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: pools: 4 pools, 137 pgs 2024-02-10T01:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: objects: 1.27k objects, 4.2 GiB 2024-02-10T01:32:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: usage: 8.7 GiB used, 261 GiB / 270 GiB avail 2024-02-10T01:32:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: pgs: 137 active+clean 2024-02-10T01:32:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: io: 2024-02-10T01:32:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client: 8.1 KiB/s rd, 3.4 KiB/s wr, 8 op/s rd, 3 op/s wr 2024-02-10T01:32:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 service dump 2024-02-10T01:32:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:{ 2024-02-10T01:32:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "epoch": 29, 2024-02-10T01:32:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "modified": "2024-02-10T01:32:13.740713+0000", 2024-02-10T01:32:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "services": { 2024-02-10T01:32:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "rbd-mirror": { 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "daemons": { 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "summary": "", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "4450": { 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_epoch": 11, 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_stamp": "2024-02-10T01:08:30.576744+0000", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "gid": 4450, 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "addr": "172.21.15.60:0/4277535184", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "metadata": { 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "arch": "x86_64", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_release": "squid", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version_short": "19.0.0-1361-g290212a1", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-02-10T01:32:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro": "centos", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_description": "CentOS Stream 9", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_version": "9", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "hostname": "smithi060", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "id": "mirror.2", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "instance_id": "4450", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_version": "5.14.0-419.el9.x86_64", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_swap_kb": "0", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_total_kb": "32494660", 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "os": "Linux" 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "task_status": {} 2024-02-10T01:32:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "4739": { 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_epoch": 15, 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_stamp": "2024-02-10T01:15:38.655330+0000", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "gid": 4739, 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "addr": "172.21.15.60:0/405173798", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "metadata": { 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "arch": "x86_64", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_release": "squid", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version_short": "19.0.0-1361-g290212a1", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro": "centos", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_description": "CentOS Stream 9", 2024-02-10T01:32:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_version": "9", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "hostname": "smithi060", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "id": "mirror.3", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "instance_id": "4739", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_version": "5.14.0-419.el9.x86_64", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_swap_kb": "0", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_total_kb": "32494660", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "os": "Linux" 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "task_status": {} 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "4754": { 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_epoch": 18, 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_stamp": "2024-02-10T01:18:18.597443+0000", 2024-02-10T01:32:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "gid": 4754, 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "addr": "172.21.15.60:0/1809266202", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "metadata": { 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "arch": "x86_64", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_release": "squid", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version_short": "19.0.0-1361-g290212a1", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro": "centos", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_description": "CentOS Stream 9", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_version": "9", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "hostname": "smithi060", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "id": "mirror.1", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "instance_id": "4754", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024", 2024-02-10T01:32:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_version": "5.14.0-419.el9.x86_64", 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_swap_kb": "0", 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_total_kb": "32494660", 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "os": "Linux" 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "task_status": {} 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "5341": { 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_epoch": 27, 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "start_stamp": "2024-02-10T01:27:05.630310+0000", 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "gid": 5341, 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "addr": "172.21.15.60:0/3534164124", 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "metadata": { 2024-02-10T01:32:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "arch": "x86_64", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_release": "squid", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "ceph_version_short": "19.0.0-1361-g290212a1", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro": "centos", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_description": "CentOS Stream 9", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "distro_version": "9", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "hostname": "smithi060", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "id": "mirror.0", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "instance_id": "5341", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "kernel_version": "5.14.0-419.el9.x86_64", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_swap_kb": "0", 2024-02-10T01:32:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "mem_total_kb": "32494660", 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "os": "Linux" 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "task_status": {} 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:} 2024-02-10T01:32:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 service status 2024-02-10T01:32:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:{ 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "rbd-mirror": { 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "4450": { 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status_stamp": "2024-02-10T01:32:05.638688+0000", 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "last_beacon": "2024-02-10T01:32:25.639610+0000", 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status": { 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_local_count\":50,\"image_remote_count\":50,\"image_warning_count\":0,\"instance_id\":\"4453\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"4455\",\"leader\":true}}" 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "4739": { 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status_stamp": "2024-02-10T01:32:13.699266+0000", 2024-02-10T01:32:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "last_beacon": "2024-02-10T01:32:28.699875+0000", 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status": { 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4742\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4745\"}}" 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "4754": { 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status_stamp": "2024-02-10T01:32:23.633321+0000", 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "last_beacon": "2024-02-10T01:32:28.633367+0000", 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status": { 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4757\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4760\"}}" 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: }, 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "5341": { 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status_stamp": "2024-02-10T01:32:10.644172+0000", 2024-02-10T01:32:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "last_beacon": "2024-02-10T01:32:25.644795+0000", 2024-02-10T01:32:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "status": { 2024-02-10T01:32:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5344\"},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5347\"}}" 2024-02-10T01:32:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: } 2024-02-10T01:32:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:} 2024-02-10T01:32:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-02-10T01:32:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:32:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror images 2024-02-10T01:32:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror images' 2024-02-10T01:32:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:NAME SIZE PARENT FMT PROT LOCK 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1@snap 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_10 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_10@snap 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11@snap 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12 128 MiB 2 2024-02-10T01:32:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_13 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_13@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_14 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_14@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_15 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_15@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_16 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_16@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_17 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_17@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_18 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_18@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_19 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_19@snap 128 MiB 2 2024-02-10T01:32:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_2 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_2@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_20 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_20@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_21 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_21@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_22 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_22@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_23 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_23@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_24 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_24@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_25 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_25@snap 128 MiB 2 2024-02-10T01:32:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_26 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_26@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_27 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_27@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_28 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_28@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_29 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_29@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_3 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_3@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_30 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_30@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_31 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_31@snap 128 MiB 2 2024-02-10T01:32:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_32 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_32@snap 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_33 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_33@snap 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_34 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_34@snap 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_35 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_36 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_37 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_38 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_39 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_4 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_4@snap 128 MiB 2 2024-02-10T01:32:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_40 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_41 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_42 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_43 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_44 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_45 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_46 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_47 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_48 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_49 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_5 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_5@snap 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_50 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_6 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_6@snap 128 MiB 2 2024-02-10T01:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_7 128 MiB 2 2024-02-10T01:32:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_7@snap 128 MiB 2 2024-02-10T01:32:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_8 128 MiB 2 2024-02-10T01:32:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_8@snap 128 MiB 2 2024-02-10T01:32:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_9 128 MiB 2 2024-02-10T01:32:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_9@snap 128 MiB 2 2024-02-10T01:32:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror mirror pool info 2024-02-10T01:32:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror mirror pool info' 2024-02-10T01:32:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2024-02-10T01:32:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mode: image 2024-02-10T01:32:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Site Name: cluster1 2024-02-10T01:32:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Peer Sites: 2024-02-10T01:32:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:UUID: 2dc2868b-b7dc-4646-98ea-74d206deca06 2024-02-10T01:32:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Name: cluster2 2024-02-10T01:32:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirror UUID: 45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Direction: rx-tx 2024-02-10T01:32:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Client: client.mirror 2024-02-10T01:32:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror mirror pool status 2024-02-10T01:32:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror mirror pool status' 2024-02-10T01:32:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2024-02-10T01:32:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:health: WARNING 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:daemon health: OK 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image health: WARNING 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:images: 50 total 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 50 starting_replay 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:DAEMONS 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4450: 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4453 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.2 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: true 2024-02-10T01:32:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4739: 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4742 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.3 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4754: 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4757 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.1 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 5341: 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 5344 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.0 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:IMAGES 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1: 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_10: 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134051840.0,"last_snapshot_bytes":134051840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528275,"remote_snapshot_timestamp":1707528275,"replay_state":"idle"} 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11: 2024-02-10T01:32:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:32:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134182912.0,"last_snapshot_bytes":134182912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528282,"remote_snapshot_timestamp":1707528282,"replay_state":"idle"} 2024-02-10T01:32:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12: 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134105088.0,"last_snapshot_bytes":134105088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528289,"remote_snapshot_timestamp":1707528289,"replay_state":"idle"} 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_13: 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134079488.0,"last_snapshot_bytes":134079488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528297,"remote_snapshot_timestamp":1707528297,"replay_state":"idle"} 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_14: 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134072320.0,"last_snapshot_bytes":134072320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528305,"remote_snapshot_timestamp":1707528305,"replay_state":"idle"} 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_15: 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528313,"remote_snapshot_timestamp":1707528313,"replay_state":"idle"} 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_16: 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528321,"remote_snapshot_timestamp":1707528321,"replay_state":"idle"} 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_17: 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134111232.0,"last_snapshot_bytes":134111232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528328,"remote_snapshot_timestamp":1707528328,"replay_state":"idle"} 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_18: 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528335,"remote_snapshot_timestamp":1707528335,"replay_state":"idle"} 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_19: 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133380096.0,"last_snapshot_bytes":133380096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528341,"remote_snapshot_timestamp":1707528341,"replay_state":"idle"} 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_2: 2024-02-10T01:32:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:32:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133945344.0,"last_snapshot_bytes":133945344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528186,"remote_snapshot_timestamp":1707528186,"replay_state":"idle"} 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_20: 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528347,"remote_snapshot_timestamp":1707528347,"replay_state":"idle"} 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_21: 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528354,"remote_snapshot_timestamp":1707528354,"replay_state":"idle"} 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_22: 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133369856.0,"last_snapshot_bytes":133369856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528361,"remote_snapshot_timestamp":1707528361,"replay_state":"idle"} 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_23: 2024-02-10T01:32:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:32:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133966848.0,"last_snapshot_bytes":133966848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528368,"remote_snapshot_timestamp":1707528368,"replay_state":"idle"} 2024-02-10T01:32:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_24: 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528375,"remote_snapshot_timestamp":1707528375,"replay_state":"idle"} 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_25: 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528381,"remote_snapshot_timestamp":1707528381,"replay_state":"idle"} 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_26: 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133741568.0,"last_snapshot_bytes":133741568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528387,"remote_snapshot_timestamp":1707528387,"replay_state":"idle"} 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_27: 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67037696.0,"last_snapshot_bytes":134075392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528393,"remote_snapshot_timestamp":1707528393,"replay_state":"idle"} 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_28: 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66970112.0,"last_snapshot_bytes":133940224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528399,"remote_snapshot_timestamp":1707528399,"replay_state":"idle"} 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_29: 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528407,"remote_snapshot_timestamp":1707528407,"replay_state":"idle"} 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_3: 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528193,"remote_snapshot_timestamp":1707528193,"replay_state":"idle"} 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_30: 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133748736.0,"last_snapshot_bytes":133748736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528415,"remote_snapshot_timestamp":1707528415,"replay_state":"idle"} 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_31: 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66904064.0,"last_snapshot_bytes":133808128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528423,"remote_snapshot_timestamp":1707528423,"replay_state":"idle"} 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_32: 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66385408.0,"last_snapshot_bytes":132770816,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528430,"remote_snapshot_timestamp":1707528430,"replay_state":"idle"} 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_33: 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":133832704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528436,"remote_snapshot_timestamp":1707528436,"replay_state":"idle"} 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_34: 2024-02-10T01:32:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:32:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134185984.0,"last_snapshot_bytes":134185984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528443,"remote_snapshot_timestamp":1707528443,"replay_state":"idle"} 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_35: 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 67ccb183-b9d9-46d5-8a34-38d09683d20e 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707527998,"remote_snapshot_timestamp":1707527998,"replay_state":"idle"} 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_36: 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: ac3ee523-253b-453f-9e13-a4d28ce2c454 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528012,"remote_snapshot_timestamp":1707528012,"replay_state":"idle"} 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_37: 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: c71c4846-b43a-41df-be57-befe30f9b331 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528023,"remote_snapshot_timestamp":1707528023,"replay_state":"idle"} 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_38: 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 0e8f3781-2b98-408e-b647-688572c90bbb 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528035,"remote_snapshot_timestamp":1707528035,"replay_state":"idle"} 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_39: 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: a70dea2d-caa0-45be-8634-f6297695c64e 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528040,"remote_snapshot_timestamp":1707528040,"replay_state":"idle"} 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_4: 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528199,"remote_snapshot_timestamp":1707528199,"replay_state":"idle"} 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_40: 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: fa535af5-2334-4635-9fbe-5c4d54ca6819 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528048,"remote_snapshot_timestamp":1707528048,"replay_state":"idle"} 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_41: 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: e8b28d26-f2b8-40d9-8c1e-01e760d124f8 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528054,"remote_snapshot_timestamp":1707528054,"replay_state":"idle"} 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_42: 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528064,"remote_snapshot_timestamp":1707528064,"replay_state":"idle"} 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_43: 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528076,"remote_snapshot_timestamp":1707528076,"replay_state":"idle"} 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_44: 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: acaf3e70-c7f2-4137-b839-6ab0448165bb 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528084,"remote_snapshot_timestamp":1707528084,"replay_state":"idle"} 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_45: 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: e5c1b9f4-c50d-41b1-be2c-c5c92fd22119 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528091,"remote_snapshot_timestamp":1707528091,"replay_state":"idle"} 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_46: 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 0b89dccc-dd99-4059-94f0-74a8c09ad67b 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528100,"remote_snapshot_timestamp":1707528100,"replay_state":"idle"} 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_47: 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 9adfb392-ed84-47b2-9257-a61dc69d11f1 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528109,"remote_snapshot_timestamp":1707528109,"replay_state":"idle"} 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_48: 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 397dba79-e127-495d-94c6-4a204e4c5269 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528120,"remote_snapshot_timestamp":1707528120,"replay_state":"idle"} 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_49: 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: c13056e5-b191-4d8c-96f9-917ad3e45ceb 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528130,"remote_snapshot_timestamp":1707528130,"replay_state":"idle"} 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_5: 2024-02-10T01:32:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:32:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528239,"remote_snapshot_timestamp":1707528239,"replay_state":"idle"} 2024-02-10T01:32:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.2 on smithi060 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:00 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:04 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_50: 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 042fec75-49d5-40e6-98a0-d964d373c5d4 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528142,"remote_snapshot_timestamp":1707528142,"replay_state":"idle"} 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_6: 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528246,"remote_snapshot_timestamp":1707528246,"replay_state":"idle"} 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_7: 2024-02-10T01:32:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:32:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":132904960.0,"last_snapshot_bytes":132904960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528254,"remote_snapshot_timestamp":1707528254,"replay_state":"idle"} 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.3 on smithi060 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:08 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_8: 2024-02-10T01:32:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:32:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134105088.0,"last_snapshot_bytes":134105088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528261,"remote_snapshot_timestamp":1707528261,"replay_state":"idle"} 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.1 on smithi060 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:18 2024-02-10T01:32:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_9: 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: global_id: 1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1707528267,"remote_snapshot_timestamp":1707528267,"replay_state":"idle"} 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:05 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: peer_sites: 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: name: cluster2 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+starting_replay 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: starting replay 2024-02-10T01:32:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: last_update: 2024-02-10 01:32:19 2024-02-10T01:32:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls 2024-02-10T01:32:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_1 info 2024-02-10T01:32:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_1 info' 2024-02-10T01:32:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_1 2024-02-10T01:32:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_1': 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10adb729b73c 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10adb729b73c 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:07 2024 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:23:03 2024 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:07 2024 2024-02-10T01:32:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:32:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_1 journal status 2024-02-10T01:32:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_1 journal status' 2024-02-10T01:32:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_1 2024-02-10T01:32:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_1 2024-02-10T01:32:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_1 snapshots 2024-02-10T01:32:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_1 snapshots' 2024-02-10T01:32:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_1 2024-02-10T01:32:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 85 snap 128 MiB Sat Feb 10 01:23:01 2024 user 2024-02-10T01:32:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 86 .mirror.non_primary.4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1.38a6727c-2bea-4dd6-8d20-cf501a67a2d8 128 MiB Sat Feb 10 01:23:02 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:87 copied) 2024-02-10T01:32:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_10 info 2024-02-10T01:32:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_10 info' 2024-02-10T01:32:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_10 2024-02-10T01:32:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_10': 2024-02-10T01:32:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165126a3d7 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165126a3d7 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:17 2024 2024-02-10T01:32:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:40 2024 2024-02-10T01:32:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:17 2024 2024-02-10T01:32:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:32:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_10 journal status 2024-02-10T01:32:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_10 journal status' 2024-02-10T01:32:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_10 2024-02-10T01:32:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_10 2024-02-10T01:32:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_10 snapshots 2024-02-10T01:32:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_10 snapshots' 2024-02-10T01:32:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_10 2024-02-10T01:32:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 103 snap 128 MiB Sat Feb 10 01:24:35 2024 user 2024-02-10T01:32:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 104 .mirror.non_primary.9471fc30-2e80-4266-99f2-6b7db30b084b.cd1dab02-5714-4c75-8271-5f4f9ad98563 128 MiB Sat Feb 10 01:24:37 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:105 copied) 2024-02-10T01:32:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_11 info 2024-02-10T01:32:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_11 info' 2024-02-10T01:32:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_11 2024-02-10T01:32:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_11': 2024-02-10T01:32:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa4d6b3ff3 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa4d6b3ff3 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:26 2024 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:47 2024 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:26 2024 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:32:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_11 journal status 2024-02-10T01:32:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_11 journal status' 2024-02-10T01:32:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_11 2024-02-10T01:32:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_11 2024-02-10T01:32:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_11 snapshots 2024-02-10T01:32:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_11 snapshots' 2024-02-10T01:32:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_11 2024-02-10T01:32:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 105 snap 128 MiB Sat Feb 10 01:24:44 2024 user 2024-02-10T01:32:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 106 .mirror.non_primary.899fe326-3145-4110-8d10-4de3c934bda0.3d0bc44e-bd60-46c4-9ca2-70ecffbd45fd 128 MiB Sat Feb 10 01:24:45 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:107 copied) 2024-02-10T01:32:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_12 info 2024-02-10T01:32:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_12 info' 2024-02-10T01:32:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_12 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_12': 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201c430258b 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201c430258b 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:39 2024 2024-02-10T01:32:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:54 2024 2024-02-10T01:32:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:39 2024 2024-02-10T01:32:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:32:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_12 journal status 2024-02-10T01:32:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_12 journal status' 2024-02-10T01:32:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_12 2024-02-10T01:32:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_12 2024-02-10T01:32:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_12 snapshots 2024-02-10T01:32:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_12 snapshots' 2024-02-10T01:32:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_12 2024-02-10T01:32:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 107 snap 128 MiB Sat Feb 10 01:24:50 2024 user 2024-02-10T01:32:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 108 .mirror.non_primary.d3c06a45-b47d-49f7-accc-1f518c871547.eb1afeab-0702-4796-aa70-376b7d2b9b33 128 MiB Sat Feb 10 01:24:51 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:109 copied) 2024-02-10T01:32:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_13 info 2024-02-10T01:32:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_13 info' 2024-02-10T01:32:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_13 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_13': 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 128631c941d5 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.128631c941d5 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:53 2024 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:01 2024 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:53 2024 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:32:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_13 journal status 2024-02-10T01:32:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_13 journal status' 2024-02-10T01:32:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_13 2024-02-10T01:32:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_13 2024-02-10T01:32:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_13 snapshots 2024-02-10T01:32:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_13 snapshots' 2024-02-10T01:32:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_13 2024-02-10T01:32:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 109 snap 128 MiB Sat Feb 10 01:24:57 2024 user 2024-02-10T01:32:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 110 .mirror.non_primary.84b62668-164a-4498-bc61-c5dc50e0965f.dd3fe7c9-d83d-4baa-92ca-a6e0e37b20d2 128 MiB Sat Feb 10 01:24:59 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:111 copied) 2024-02-10T01:32:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_14 info 2024-02-10T01:32:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_14 info' 2024-02-10T01:32:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_14 2024-02-10T01:32:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_14': 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1286a0cf4358 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1286a0cf4358 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:05 2024 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:09 2024 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:05 2024 2024-02-10T01:32:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:32:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_14 journal status 2024-02-10T01:32:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_14 journal status' 2024-02-10T01:32:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_14 2024-02-10T01:32:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_14 2024-02-10T01:32:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_14 snapshots 2024-02-10T01:32:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_14 snapshots' 2024-02-10T01:32:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_14 2024-02-10T01:32:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 111 snap 128 MiB Sat Feb 10 01:25:06 2024 user 2024-02-10T01:32:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 112 .mirror.non_primary.5b5e467c-0819-4724-bbd1-15702a5c84ef.952c9e6f-07fe-43d0-b88c-343881aa62a9 128 MiB Sat Feb 10 01:25:07 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:113 copied) 2024-02-10T01:32:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_15 info 2024-02-10T01:32:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_15 info' 2024-02-10T01:32:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_15 2024-02-10T01:32:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_15': 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201eb3deb91 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201eb3deb91 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:18 2024 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:17 2024 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:18 2024 2024-02-10T01:32:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:32:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_15 journal status 2024-02-10T01:32:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_15 journal status' 2024-02-10T01:32:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_15 2024-02-10T01:32:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_15 2024-02-10T01:32:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_15 snapshots 2024-02-10T01:32:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_15 snapshots' 2024-02-10T01:32:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_15 2024-02-10T01:32:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 113 snap 128 MiB Sat Feb 10 01:25:14 2024 user 2024-02-10T01:32:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 114 .mirror.non_primary.7514c22c-dff0-4292-b368-8aba11dec642.17c4c273-cb79-4597-a934-63f90f460b46 128 MiB Sat Feb 10 01:25:15 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:115 copied) 2024-02-10T01:32:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_16 info' 2024-02-10T01:32:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_16 info 2024-02-10T01:32:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_16 2024-02-10T01:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_16': 2024-02-10T01:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12863cb0a88b 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12863cb0a88b 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:30 2024 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:25 2024 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:30 2024 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_16 journal status 2024-02-10T01:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_16 journal status' 2024-02-10T01:32:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_16 2024-02-10T01:32:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_16 2024-02-10T01:32:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_16 snapshots 2024-02-10T01:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_16 snapshots' 2024-02-10T01:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_16 2024-02-10T01:32:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 115 snap 128 MiB Sat Feb 10 01:25:22 2024 user 2024-02-10T01:32:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 116 .mirror.non_primary.93bbc219-821b-43e5-a754-4712702c247e.1ae4ebe8-8b5d-4b27-9261-f47a1f0aff7e 128 MiB Sat Feb 10 01:25:23 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:117 copied) 2024-02-10T01:32:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_17 info 2024-02-10T01:32:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_17 info' 2024-02-10T01:32:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_17 2024-02-10T01:32:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_17': 2024-02-10T01:32:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165f6384a7a 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165f6384a7a 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:39 2024 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:32 2024 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:39 2024 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:32:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_17 journal status 2024-02-10T01:32:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_17 journal status' 2024-02-10T01:32:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_17 2024-02-10T01:32:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_17 2024-02-10T01:32:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_17 snapshots 2024-02-10T01:32:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_17 snapshots' 2024-02-10T01:32:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_17 2024-02-10T01:32:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 117 snap 128 MiB Sat Feb 10 01:25:29 2024 user 2024-02-10T01:32:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 118 .mirror.non_primary.af0461fa-e6c8-4e21-90df-d164bb810e1c.c281d980-49e8-4369-9d3d-53c219800c22 128 MiB Sat Feb 10 01:25:30 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:119 copied) 2024-02-10T01:32:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_18 info 2024-02-10T01:32:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_18 info' 2024-02-10T01:32:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_18 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_18': 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aafc5876ef 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aafc5876ef 2024-02-10T01:32:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:48 2024 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:38 2024 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:48 2024 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:32:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_18 journal status 2024-02-10T01:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_18 journal status' 2024-02-10T01:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_18 2024-02-10T01:32:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_18 2024-02-10T01:32:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_18 snapshots 2024-02-10T01:32:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_18 snapshots' 2024-02-10T01:32:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_18 2024-02-10T01:32:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 119 snap 128 MiB Sat Feb 10 01:25:36 2024 user 2024-02-10T01:32:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 120 .mirror.non_primary.a51c283d-16da-4e8f-a80b-33be6eb1572b.219667dc-e8c4-4e47-80ac-8718a66ea42c 128 MiB Sat Feb 10 01:25:37 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:121 copied) 2024-02-10T01:32:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_19 info 2024-02-10T01:32:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_19 info' 2024-02-10T01:32:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_19 2024-02-10T01:32:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_19': 2024-02-10T01:32:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201a7e8981b 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201a7e8981b 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:16:55 2024 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:44 2024 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:16:55 2024 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:32:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_19 journal status 2024-02-10T01:32:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_19 journal status' 2024-02-10T01:32:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_19 2024-02-10T01:32:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_19 2024-02-10T01:32:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_19 snapshots 2024-02-10T01:32:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_19 snapshots' 2024-02-10T01:32:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_19 2024-02-10T01:32:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 121 snap 128 MiB Sat Feb 10 01:25:42 2024 user 2024-02-10T01:32:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 122 .mirror.non_primary.cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09.fbc9488d-9476-4f43-8391-ebdc15e442bc 128 MiB Sat Feb 10 01:25:43 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:123 copied) 2024-02-10T01:32:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_2 info 2024-02-10T01:32:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_2 info' 2024-02-10T01:32:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_2 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_2': 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165e057a8a9 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165e057a8a9 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:14 2024 2024-02-10T01:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:23:10 2024 2024-02-10T01:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:14 2024 2024-02-10T01:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_2 journal status 2024-02-10T01:32:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_2 journal status' 2024-02-10T01:32:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_2 2024-02-10T01:32:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_2 2024-02-10T01:32:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_2 snapshots 2024-02-10T01:32:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_2 snapshots' 2024-02-10T01:32:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_2 2024-02-10T01:32:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 87 snap 128 MiB Sat Feb 10 01:23:07 2024 user 2024-02-10T01:32:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 88 .mirror.non_primary.69e299aa-5815-4202-b2c2-9fd34a982ad9.05a5c335-94bb-46df-90d5-ab9d3afb7d9f 128 MiB Sat Feb 10 01:23:08 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:89 copied) 2024-02-10T01:32:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_20 info 2024-02-10T01:32:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_20 info' 2024-02-10T01:32:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_20 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_20': 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12863cb65f06 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12863cb65f06 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:03 2024 2024-02-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:51 2024 2024-02-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:03 2024 2024-02-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_20 journal status 2024-02-10T01:32:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_20 journal status' 2024-02-10T01:32:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_20 2024-02-10T01:32:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_20 2024-02-10T01:32:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_20 snapshots 2024-02-10T01:32:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_20 snapshots' 2024-02-10T01:32:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_20 2024-02-10T01:32:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 123 snap 128 MiB Sat Feb 10 01:25:48 2024 user 2024-02-10T01:32:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 124 .mirror.non_primary.565aeb89-9b20-428e-8ba0-8c8a401c0dc0.cefd021f-0b62-4f83-87b6-f256cddde589 128 MiB Sat Feb 10 01:25:49 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:125 copied) 2024-02-10T01:32:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_21 info 2024-02-10T01:32:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_21 info' 2024-02-10T01:32:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_21 2024-02-10T01:32:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_21': 2024-02-10T01:32:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165d7c7ae39 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165d7c7ae39 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:14 2024 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:25:58 2024 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:14 2024 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:32:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_21 journal status 2024-02-10T01:32:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_21 journal status' 2024-02-10T01:32:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_21 2024-02-10T01:32:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_21 2024-02-10T01:32:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_21 snapshots 2024-02-10T01:32:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_21 snapshots' 2024-02-10T01:32:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_21 2024-02-10T01:32:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 125 snap 128 MiB Sat Feb 10 01:25:55 2024 user 2024-02-10T01:32:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 126 .mirror.non_primary.bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e.3c8a9c58-239f-4f10-90ab-c7af61aeaaab 128 MiB Sat Feb 10 01:25:56 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:127 copied) 2024-02-10T01:32:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_22 info 2024-02-10T01:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_22 info' 2024-02-10T01:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_22 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_22': 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aafdc11b2c 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aafdc11b2c 2024-02-10T01:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:28 2024 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:05 2024 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:28 2024 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:32:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_22 journal status 2024-02-10T01:32:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_22 journal status' 2024-02-10T01:32:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_22 2024-02-10T01:32:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_22 2024-02-10T01:32:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_22 snapshots 2024-02-10T01:32:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_22 snapshots' 2024-02-10T01:32:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_22 2024-02-10T01:32:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 127 snap 128 MiB Sat Feb 10 01:26:02 2024 user 2024-02-10T01:32:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 128 .mirror.non_primary.b2460f66-10b1-46c5-80cc-386b73158112.5f153900-4e8f-4e6e-ad77-a6ef4a80c5d7 128 MiB Sat Feb 10 01:26:03 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:129 copied) 2024-02-10T01:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_23 info 2024-02-10T01:32:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_23 info' 2024-02-10T01:32:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_23 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_23': 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201676a5a41 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201676a5a41 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:38 2024 2024-02-10T01:32:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:12 2024 2024-02-10T01:32:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:38 2024 2024-02-10T01:32:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:32:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_23 journal status 2024-02-10T01:32:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_23 journal status' 2024-02-10T01:32:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_23 2024-02-10T01:32:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_23 2024-02-10T01:32:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_23 snapshots 2024-02-10T01:32:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_23 snapshots' 2024-02-10T01:32:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_23 2024-02-10T01:32:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 129 snap 128 MiB Sat Feb 10 01:26:09 2024 user 2024-02-10T01:32:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 130 .mirror.non_primary.a6a210df-3fd5-4b61-aab9-877c0b10dc4a.7c235d18-fb23-4ec9-b850-ba17099428f6 128 MiB Sat Feb 10 01:26:10 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:131 copied) 2024-02-10T01:32:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_24 info 2024-02-10T01:32:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_24 info' 2024-02-10T01:32:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_24 2024-02-10T01:32:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_24': 2024-02-10T01:32:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1286c08399a4 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1286c08399a4 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:17:51 2024 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:18 2024 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:17:51 2024 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:32:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_24 journal status 2024-02-10T01:32:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_24 journal status' 2024-02-10T01:32:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_24 2024-02-10T01:32:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_24 2024-02-10T01:32:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_24 snapshots' 2024-02-10T01:32:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_24 snapshots 2024-02-10T01:32:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_24 2024-02-10T01:32:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 131 snap 128 MiB Sat Feb 10 01:26:16 2024 user 2024-02-10T01:32:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 132 .mirror.non_primary.98913dec-0f18-4c81-b449-d8572d68cfe9.21819167-4a11-433a-b580-3868434e25af 128 MiB Sat Feb 10 01:26:17 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:133 copied) 2024-02-10T01:32:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_25 info' 2024-02-10T01:32:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_25 info 2024-02-10T01:32:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_25 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_25': 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165fd99f663 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165fd99f663 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:01 2024 2024-02-10T01:32:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:24 2024 2024-02-10T01:32:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:01 2024 2024-02-10T01:32:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:32:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_25 journal status 2024-02-10T01:32:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_25 journal status' 2024-02-10T01:32:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_25 2024-02-10T01:32:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_25 2024-02-10T01:32:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_25 snapshots 2024-02-10T01:32:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_25 snapshots' 2024-02-10T01:32:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_25 2024-02-10T01:32:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 133 snap 128 MiB Sat Feb 10 01:26:22 2024 user 2024-02-10T01:32:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 134 .mirror.non_primary.4337aed3-d9ec-4cea-a99c-5193193a5e37.f5ef4b3b-134f-46ce-bf84-8df176dd6aa2 128 MiB Sat Feb 10 01:26:23 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:135 copied) 2024-02-10T01:32:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_26 info 2024-02-10T01:32:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_26 info' 2024-02-10T01:32:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_26 2024-02-10T01:32:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_26': 2024-02-10T01:32:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa684a031a 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa684a031a 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:09 2024 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:30 2024 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:09 2024 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:32:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_26 journal status 2024-02-10T01:32:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_26 journal status' 2024-02-10T01:32:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_26 2024-02-10T01:32:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_26 2024-02-10T01:32:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_26 snapshots 2024-02-10T01:32:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_26 snapshots' 2024-02-10T01:32:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_26 2024-02-10T01:32:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 135 snap 128 MiB Sat Feb 10 01:26:28 2024 user 2024-02-10T01:32:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 136 .mirror.non_primary.f0fc7c92-9985-462d-8426-1d934c12daa6.dfc673db-aaa7-41fb-b962-91cdb174224a 128 MiB Sat Feb 10 01:26:29 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:137 copied) 2024-02-10T01:32:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_27 info' 2024-02-10T01:32:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_27 info 2024-02-10T01:32:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_27 2024-02-10T01:32:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_27': 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295f903f63f 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295f903f63f 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:45 2024 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:36 2024 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:45 2024 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:32:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_27 journal status 2024-02-10T01:32:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_27 journal status' 2024-02-10T01:32:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_27 2024-02-10T01:32:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_27 2024-02-10T01:32:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_27 snapshots 2024-02-10T01:32:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_27 snapshots' 2024-02-10T01:32:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_27 2024-02-10T01:32:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 137 snap 128 MiB Sat Feb 10 01:26:34 2024 user 2024-02-10T01:32:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 138 .mirror.non_primary.c9d8a153-d84b-4460-b3b0-8069daffb69f.617383f5-0b08-44f2-bcf3-53da33a307a1 128 MiB Sat Feb 10 01:26:35 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:139 copied) 2024-02-10T01:32:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_28 info 2024-02-10T01:32:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_28 info' 2024-02-10T01:32:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_28 2024-02-10T01:32:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_28': 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295c4ad4b3e 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295c4ad4b3e 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:34 2024 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:44 2024 2024-02-10T01:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:34 2024 2024-02-10T01:32:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:32:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_28 journal status 2024-02-10T01:32:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_28 journal status' 2024-02-10T01:32:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_28 2024-02-10T01:32:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_28 2024-02-10T01:32:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_28 snapshots 2024-02-10T01:32:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_28 snapshots' 2024-02-10T01:32:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_28 2024-02-10T01:32:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 139 snap 128 MiB Sat Feb 10 01:26:41 2024 user 2024-02-10T01:32:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 140 .mirror.non_primary.b77ba965-f52f-4cd2-8014-4c431ed84c93.657a1a9b-09e9-4e16-8298-1dfb9843ccba 128 MiB Sat Feb 10 01:26:42 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:141 copied) 2024-02-10T01:32:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_29 info 2024-02-10T01:32:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_29 info' 2024-02-10T01:32:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_29 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_29': 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 116597bb13ad 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.116597bb13ad 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:18:46 2024 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:51 2024 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:18:46 2024 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_29 journal status 2024-02-10T01:32:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_29 journal status' 2024-02-10T01:32:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_29 2024-02-10T01:32:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_29 2024-02-10T01:32:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_29 snapshots 2024-02-10T01:32:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_29 snapshots' 2024-02-10T01:32:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_29 2024-02-10T01:32:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 141 snap 128 MiB Sat Feb 10 01:26:48 2024 user 2024-02-10T01:32:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 142 .mirror.non_primary.5b16c1c4-67da-455b-9baf-d531bb015e83.f6a757f7-70a5-4f1e-aff1-d7cd012061fe 128 MiB Sat Feb 10 01:26:49 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:143 copied) 2024-02-10T01:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_3 info 2024-02-10T01:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_3 info' 2024-02-10T01:32:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_3 2024-02-10T01:32:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_3': 2024-02-10T01:32:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa200b883a 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa200b883a 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:23 2024 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:23:16 2024 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:23 2024 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:32:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_3 journal status 2024-02-10T01:32:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_3 journal status' 2024-02-10T01:32:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_3 2024-02-10T01:32:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_3 2024-02-10T01:32:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_3 snapshots 2024-02-10T01:32:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_3 snapshots' 2024-02-10T01:32:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_3 2024-02-10T01:32:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 89 snap 128 MiB Sat Feb 10 01:23:14 2024 user 2024-02-10T01:32:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 90 .mirror.non_primary.4f363a97-9a20-4913-a174-4d1fab8a425e.2dc3e3fc-3d3f-4256-b55b-0e20fc7df45d 128 MiB Sat Feb 10 01:23:15 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:91 copied) 2024-02-10T01:32:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_30 info 2024-02-10T01:32:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_30 info' 2024-02-10T01:32:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_30 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_30': 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa8142e435 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa8142e435 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:01 2024 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:26:59 2024 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:01 2024 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:32:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_30 journal status 2024-02-10T01:32:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_30 journal status' 2024-02-10T01:32:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_30 2024-02-10T01:32:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_30 2024-02-10T01:32:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_30 snapshots 2024-02-10T01:32:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_30 snapshots' 2024-02-10T01:32:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_30 2024-02-10T01:32:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 143 snap 128 MiB Sat Feb 10 01:26:56 2024 user 2024-02-10T01:32:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 144 .mirror.non_primary.c47c4ec6-f47f-432b-9a07-7b157c27bcb2.ce1373b6-57af-43eb-8dcf-714e9670cc5d 128 MiB Sat Feb 10 01:26:57 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:145 copied) 2024-02-10T01:32:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_31 info 2024-02-10T01:32:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_31 info' 2024-02-10T01:32:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_31 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_31': 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12863cc0137b 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12863cc0137b 2024-02-10T01:32:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:13 2024 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:27:07 2024 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:13 2024 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:32:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_31 journal status 2024-02-10T01:32:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_31 journal status' 2024-02-10T01:32:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_31 2024-02-10T01:32:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_31 2024-02-10T01:32:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_31 snapshots 2024-02-10T01:32:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_31 snapshots' 2024-02-10T01:32:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_31 2024-02-10T01:32:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 145 snap 128 MiB Sat Feb 10 01:27:04 2024 user 2024-02-10T01:32:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 146 .mirror.non_primary.d2226f23-857f-451b-8d8e-d6fba2fb3cc8.5a54bf77-53f9-4b65-9d56-40d28ab72f15 128 MiB Sat Feb 10 01:27:05 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:147 copied) 2024-02-10T01:32:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_32 info 2024-02-10T01:32:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_32 info' 2024-02-10T01:32:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_32 2024-02-10T01:32:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_32': 2024-02-10T01:32:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12956af0d92f 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12956af0d92f 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:27 2024 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:27:12 2024 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:27 2024 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:32:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_32 journal status 2024-02-10T01:32:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_32 journal status' 2024-02-10T01:32:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_32 2024-02-10T01:32:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_32 2024-02-10T01:32:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_32 snapshots 2024-02-10T01:32:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_32 snapshots' 2024-02-10T01:32:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_32 2024-02-10T01:32:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 147 snap 128 MiB Sat Feb 10 01:27:10 2024 user 2024-02-10T01:32:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 148 .mirror.non_primary.cfcbfcff-3760-4d2b-977c-34a55bb67b26.3603e258-6ec0-42e4-b1f7-05f61fa3c263 128 MiB Sat Feb 10 01:27:11 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:149 copied) 2024-02-10T01:32:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_33 info 2024-02-10T01:32:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_33 info' 2024-02-10T01:32:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_33 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_33': 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 116556c1ef13 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.116556c1ef13 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:34 2024 2024-02-10T01:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:27:20 2024 2024-02-10T01:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:34 2024 2024-02-10T01:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_33 journal status 2024-02-10T01:32:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_33 journal status' 2024-02-10T01:32:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_33 2024-02-10T01:32:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_33 2024-02-10T01:32:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_33 snapshots 2024-02-10T01:32:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_33 snapshots' 2024-02-10T01:32:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_33 2024-02-10T01:32:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 149 snap 128 MiB Sat Feb 10 01:27:17 2024 user 2024-02-10T01:32:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 150 .mirror.non_primary.eaa78a82-7037-4e22-a10c-35341fcf6a99.f550f465-fec5-4061-8809-2a360aec6f03 128 MiB Sat Feb 10 01:27:18 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:151 copied) 2024-02-10T01:32:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_34 info 2024-02-10T01:32:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_34 info' 2024-02-10T01:32:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_34 2024-02-10T01:32:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_34': 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 129f93ae6656 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.129f93ae6656 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:19:49 2024 2024-02-10T01:32:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:19:49 2024 2024-02-10T01:32:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:19:49 2024 2024-02-10T01:32:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:32:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_34 journal status 2024-02-10T01:32:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_34 journal status' 2024-02-10T01:32:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_34 2024-02-10T01:32:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_34 2024-02-10T01:32:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_34 snapshots 2024-02-10T01:32:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_34 snapshots' 2024-02-10T01:32:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_34 2024-02-10T01:32:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 151 snap 128 MiB Sat Feb 10 01:27:24 2024 user 2024-02-10T01:32:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 152 .mirror.non_primary.8569f147-1a99-448c-bc58-d23cbda9ef18.803887a1-dcbd-476b-9df3-ef9dc01a4b3a 128 MiB Sat Feb 10 01:27:25 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:153 copied) 2024-02-10T01:32:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_35 info 2024-02-10T01:32:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_35 info' 2024-02-10T01:32:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_35 2024-02-10T01:32:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_35': 2024-02-10T01:32:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 129fca21d98e 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.129fca21d98e 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:00 2024 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:00 2024 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:00 2024 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 67ccb183-b9d9-46d5-8a34-38d09683d20e 2024-02-10T01:32:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_35 journal status 2024-02-10T01:32:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_35 journal status' 2024-02-10T01:32:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_35 2024-02-10T01:32:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_35 2024-02-10T01:32:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_35 snapshots 2024-02-10T01:32:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_35 snapshots' 2024-02-10T01:32:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_35 2024-02-10T01:32:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 69 .mirror.non_primary.67ccb183-b9d9-46d5-8a34-38d09683d20e.1ccbf78c-d43d-4024-84c4-4e99feb7866d 128 MiB Sat Feb 10 01:20:01 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:70 copied) 2024-02-10T01:32:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_36 info 2024-02-10T01:32:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_36 info' 2024-02-10T01:32:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_36 2024-02-10T01:32:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_36': 2024-02-10T01:32:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 129fd3f379e7 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.129fd3f379e7 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:14 2024 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:14 2024 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:14 2024 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: ac3ee523-253b-453f-9e13-a4d28ce2c454 2024-02-10T01:32:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_36 journal status 2024-02-10T01:32:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_36 journal status' 2024-02-10T01:32:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_36 2024-02-10T01:32:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_36 2024-02-10T01:32:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_36 snapshots 2024-02-10T01:32:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_36 snapshots' 2024-02-10T01:32:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_36 2024-02-10T01:32:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 70 .mirror.non_primary.ac3ee523-253b-453f-9e13-a4d28ce2c454.be1b2956-93b9-4d96-b16d-709ef1bc3593 128 MiB Sat Feb 10 01:20:14 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:71 copied) 2024-02-10T01:32:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_37 info 2024-02-10T01:32:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_37 info' 2024-02-10T01:32:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_37 2024-02-10T01:32:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_37': 2024-02-10T01:32:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11659dd9ff1 2024-02-10T01:32:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11659dd9ff1 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:25 2024 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:25 2024 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:25 2024 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c71c4846-b43a-41df-be57-befe30f9b331 2024-02-10T01:32:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_37 journal status 2024-02-10T01:32:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_37 journal status' 2024-02-10T01:32:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_37 2024-02-10T01:32:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_37 2024-02-10T01:32:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_37 snapshots 2024-02-10T01:32:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_37 snapshots' 2024-02-10T01:32:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_37 2024-02-10T01:32:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 71 .mirror.non_primary.c71c4846-b43a-41df-be57-befe30f9b331.7900cf22-b5a9-4f46-b072-d5a841c42bdd 128 MiB Sat Feb 10 01:20:26 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:72 copied) 2024-02-10T01:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_38 info 2024-02-10T01:32:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_38 info' 2024-02-10T01:32:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_38 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_38': 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1286a340854d 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1286a340854d 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:37 2024 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:37 2024 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:37 2024 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0e8f3781-2b98-408e-b647-688572c90bbb 2024-02-10T01:32:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_38 journal status 2024-02-10T01:32:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_38 journal status' 2024-02-10T01:32:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_38 2024-02-10T01:32:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_38 2024-02-10T01:32:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_38 snapshots 2024-02-10T01:32:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_38 snapshots' 2024-02-10T01:32:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_38 2024-02-10T01:32:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 72 .mirror.non_primary.0e8f3781-2b98-408e-b647-688572c90bbb.839169b7-6102-4939-940b-01c45fec2087 128 MiB Sat Feb 10 01:20:37 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:73 copied) 2024-02-10T01:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_39 info 2024-02-10T01:32:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_39 info' 2024-02-10T01:32:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_39 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_39': 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295f6cf6cec 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295f6cf6cec 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:42 2024 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:42 2024 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:42 2024 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a70dea2d-caa0-45be-8634-f6297695c64e 2024-02-10T01:32:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_39 journal status 2024-02-10T01:32:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_39 journal status' 2024-02-10T01:32:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_39 2024-02-10T01:32:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_39 2024-02-10T01:32:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_39 snapshots 2024-02-10T01:32:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_39 snapshots' 2024-02-10T01:32:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_39 2024-02-10T01:32:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 73 .mirror.non_primary.a70dea2d-caa0-45be-8634-f6297695c64e.b30a2af5-e042-46cb-8e85-e0c7a5d9c1ff 128 MiB Sat Feb 10 01:20:42 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:74 copied) 2024-02-10T01:32:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_4 info 2024-02-10T01:32:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_4 info' 2024-02-10T01:32:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_4 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_4': 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12012cb6b807 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12012cb6b807 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:31 2024 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:23:22 2024 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:31 2024 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:32:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_4 journal status 2024-02-10T01:32:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_4 journal status' 2024-02-10T01:32:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_4 2024-02-10T01:32:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_4 2024-02-10T01:32:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_4 snapshots 2024-02-10T01:32:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_4 snapshots' 2024-02-10T01:32:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_4 2024-02-10T01:32:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 91 snap 128 MiB Sat Feb 10 01:23:20 2024 user 2024-02-10T01:32:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 92 .mirror.non_primary.5173765f-eddd-4599-883b-3e3ec2eb0f2c.59bde6b9-fdc7-4d84-808e-99463fab83a2 128 MiB Sat Feb 10 01:23:21 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:93 copied) 2024-02-10T01:32:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_40 info 2024-02-10T01:32:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_40 info' 2024-02-10T01:32:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_40 2024-02-10T01:32:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_40': 2024-02-10T01:32:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 129fedb423e8 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.129fedb423e8 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:50 2024 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:50 2024 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:50 2024 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: fa535af5-2334-4635-9fbe-5c4d54ca6819 2024-02-10T01:32:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_40 journal status 2024-02-10T01:32:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_40 journal status' 2024-02-10T01:32:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_40 2024-02-10T01:32:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_40 2024-02-10T01:32:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_40 snapshots 2024-02-10T01:32:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_40 snapshots' 2024-02-10T01:32:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_40 2024-02-10T01:32:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 74 .mirror.non_primary.fa535af5-2334-4635-9fbe-5c4d54ca6819.1236ea7d-4376-4783-a4ea-82fe6aa43193 128 MiB Sat Feb 10 01:20:50 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:75 copied) 2024-02-10T01:32:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_41 info' 2024-02-10T01:32:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_41 info 2024-02-10T01:32:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_41 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_41': 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 116531e8ed38 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.116531e8ed38 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:20:56 2024 2024-02-10T01:32:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:20:56 2024 2024-02-10T01:32:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:20:56 2024 2024-02-10T01:32:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e8b28d26-f2b8-40d9-8c1e-01e760d124f8 2024-02-10T01:32:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_41 journal status 2024-02-10T01:32:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_41 journal status' 2024-02-10T01:32:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_41 2024-02-10T01:32:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_41 2024-02-10T01:32:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_41 snapshots 2024-02-10T01:32:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_41 snapshots' 2024-02-10T01:32:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_41 2024-02-10T01:32:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 75 .mirror.non_primary.e8b28d26-f2b8-40d9-8c1e-01e760d124f8.12af1942-6d35-4445-ac9c-44961c278a70 128 MiB Sat Feb 10 01:20:57 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:76 copied) 2024-02-10T01:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_42 info 2024-02-10T01:32:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_42 info' 2024-02-10T01:32:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_42 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_42': 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12869d84033c 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12869d84033c 2024-02-10T01:32:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:06 2024 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:06 2024 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:06 2024 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae 2024-02-10T01:32:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_42 journal status 2024-02-10T01:32:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_42 journal status' 2024-02-10T01:32:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_42 2024-02-10T01:32:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_42 2024-02-10T01:32:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_42 snapshots 2024-02-10T01:32:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_42 snapshots' 2024-02-10T01:32:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_42 2024-02-10T01:32:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 76 .mirror.non_primary.b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae.ed31e856-ea1c-4223-8034-b27c523b1ff5 128 MiB Sat Feb 10 01:21:07 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:77 copied) 2024-02-10T01:32:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_43 info' 2024-02-10T01:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_43 info 2024-02-10T01:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_43 2024-02-10T01:32:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_43': 2024-02-10T01:32:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12a95671e780 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12a95671e780 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:18 2024 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:18 2024 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:18 2024 2024-02-10T01:32:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb 2024-02-10T01:32:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_43 journal status 2024-02-10T01:32:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_43 journal status' 2024-02-10T01:32:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_43 2024-02-10T01:32:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_43 2024-02-10T01:32:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_43 snapshots 2024-02-10T01:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_43 snapshots' 2024-02-10T01:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_43 2024-02-10T01:32:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 77 .mirror.non_primary.e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb.c72b0733-eae0-43de-96cb-812c307ebdc0 128 MiB Sat Feb 10 01:21:19 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:78 copied) 2024-02-10T01:32:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_44 info 2024-02-10T01:32:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_44 info' 2024-02-10T01:32:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_44 2024-02-10T01:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_44': 2024-02-10T01:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12a9166c4d99 2024-02-10T01:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12a9166c4d99 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:26 2024 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:26 2024 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:26 2024 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: acaf3e70-c7f2-4137-b839-6ab0448165bb 2024-02-10T01:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_44 journal status 2024-02-10T01:32:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_44 journal status' 2024-02-10T01:32:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_44 2024-02-10T01:32:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_44 2024-02-10T01:32:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_44 snapshots 2024-02-10T01:32:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_44 snapshots' 2024-02-10T01:32:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_44 2024-02-10T01:32:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 78 .mirror.non_primary.acaf3e70-c7f2-4137-b839-6ab0448165bb.ee150303-3c65-43c2-b37f-ec2f46074f1c 128 MiB Sat Feb 10 01:21:27 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:79 copied) 2024-02-10T01:32:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_45 info 2024-02-10T01:32:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_45 info' 2024-02-10T01:32:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_45 2024-02-10T01:32:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_45': 2024-02-10T01:32:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165abab50ed 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165abab50ed 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:33 2024 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:33 2024 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:33 2024 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e5c1b9f4-c50d-41b1-be2c-c5c92fd22119 2024-02-10T01:32:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_45 journal status 2024-02-10T01:32:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_45 journal status' 2024-02-10T01:32:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_45 2024-02-10T01:32:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_45 2024-02-10T01:32:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_45 snapshots 2024-02-10T01:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_45 snapshots' 2024-02-10T01:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_45 2024-02-10T01:32:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 79 .mirror.non_primary.e5c1b9f4-c50d-41b1-be2c-c5c92fd22119.3cbe6037-06da-4b65-b313-ab7a8f00389c 128 MiB Sat Feb 10 01:21:34 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:80 copied) 2024-02-10T01:32:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_46 info 2024-02-10T01:32:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_46 info' 2024-02-10T01:32:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_46 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_46': 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1286bb7eff82 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1286bb7eff82 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:42 2024 2024-02-10T01:32:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:42 2024 2024-02-10T01:32:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:42 2024 2024-02-10T01:32:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0b89dccc-dd99-4059-94f0-74a8c09ad67b 2024-02-10T01:32:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_46 journal status 2024-02-10T01:32:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_46 journal status' 2024-02-10T01:32:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_46 2024-02-10T01:32:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_46 2024-02-10T01:32:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_46 snapshots 2024-02-10T01:32:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_46 snapshots' 2024-02-10T01:32:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_46 2024-02-10T01:32:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 80 .mirror.non_primary.0b89dccc-dd99-4059-94f0-74a8c09ad67b.c3034835-0cc2-49e6-9587-f8325de5bc46 128 MiB Sat Feb 10 01:21:43 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:81 copied) 2024-02-10T01:32:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_47 info' 2024-02-10T01:32:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_47 info 2024-02-10T01:32:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_47 2024-02-10T01:32:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_47': 2024-02-10T01:32:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1295187d0eef 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1295187d0eef 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:21:51 2024 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:21:51 2024 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:21:51 2024 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9adfb392-ed84-47b2-9257-a61dc69d11f1 2024-02-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_47 journal status 2024-02-10T01:32:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_47 journal status' 2024-02-10T01:32:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_47 2024-02-10T01:32:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_47 2024-02-10T01:32:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_47 snapshots 2024-02-10T01:32:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_47 snapshots' 2024-02-10T01:32:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_47 2024-02-10T01:32:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 81 .mirror.non_primary.9adfb392-ed84-47b2-9257-a61dc69d11f1.b94669db-2e38-4158-859e-60b210748758 128 MiB Sat Feb 10 01:21:51 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:82 copied) 2024-02-10T01:32:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_48 info' 2024-02-10T01:32:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_48 info 2024-02-10T01:32:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_48 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_48': 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12a98f768784 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12a98f768784 2024-02-10T01:32:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:22:02 2024 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:22:02 2024 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:22:02 2024 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 397dba79-e127-495d-94c6-4a204e4c5269 2024-02-10T01:32:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_48 journal status 2024-02-10T01:32:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_48 journal status' 2024-02-10T01:32:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_48 2024-02-10T01:32:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_48 2024-02-10T01:32:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_48 snapshots 2024-02-10T01:32:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_48 snapshots' 2024-02-10T01:32:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_48 2024-02-10T01:32:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 82 .mirror.non_primary.397dba79-e127-495d-94c6-4a204e4c5269.b11de918-f01e-4898-a33c-af5bcd010da9 128 MiB Sat Feb 10 01:22:02 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:83 copied) 2024-02-10T01:32:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_49 info' 2024-02-10T01:32:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_49 info 2024-02-10T01:32:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_49 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_49': 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165c1fa59e9 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165c1fa59e9 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:22:12 2024 2024-02-10T01:32:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:22:12 2024 2024-02-10T01:32:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:22:12 2024 2024-02-10T01:32:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c13056e5-b191-4d8c-96f9-917ad3e45ceb 2024-02-10T01:32:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_49 journal status 2024-02-10T01:32:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_49 journal status' 2024-02-10T01:32:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_49 2024-02-10T01:32:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_49 2024-02-10T01:32:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_49 snapshots 2024-02-10T01:32:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_49 snapshots' 2024-02-10T01:32:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_49 2024-02-10T01:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 83 .mirror.non_primary.c13056e5-b191-4d8c-96f9-917ad3e45ceb.58cfd1da-7c26-4202-ba30-5f4deaaa739c 128 MiB Sat Feb 10 01:22:13 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:84 copied) 2024-02-10T01:32:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_5 info 2024-02-10T01:32:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_5 info' 2024-02-10T01:32:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_5 2024-02-10T01:32:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_5': 2024-02-10T01:32:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10ad417595b7 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10ad417595b7 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:38 2024 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:03 2024 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:38 2024 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:32:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_5 journal status 2024-02-10T01:32:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_5 journal status' 2024-02-10T01:32:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_5 2024-02-10T01:32:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_5 2024-02-10T01:32:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_5 snapshots 2024-02-10T01:32:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_5 snapshots' 2024-02-10T01:32:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_5 2024-02-10T01:32:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 93 snap 128 MiB Sat Feb 10 01:24:00 2024 user 2024-02-10T01:32:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 94 .mirror.non_primary.278e9f1e-33e4-4605-97dc-7d7e289de2fd.d354b328-2cf1-4823-8867-9d0f111e9f87 128 MiB Sat Feb 10 01:24:01 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:95 copied) 2024-02-10T01:32:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_50 info 2024-02-10T01:32:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_50 info' 2024-02-10T01:32:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_50 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_50': 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 12b8cf703310 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.12b8cf703310 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:22:24 2024 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:22:24 2024 2024-02-10T01:32:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:22:24 2024 2024-02-10T01:32:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 042fec75-49d5-40e6-98a0-d964d373c5d4 2024-02-10T01:32:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_50 journal status 2024-02-10T01:32:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_50 journal status' 2024-02-10T01:32:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_50 2024-02-10T01:32:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_50 2024-02-10T01:32:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_50 snapshots 2024-02-10T01:32:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_50 snapshots' 2024-02-10T01:32:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_50 2024-02-10T01:32:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 84 .mirror.non_primary.042fec75-49d5-40e6-98a0-d964d373c5d4.ab7f12ad-59f3-467d-8b49-98d4c177a66f 128 MiB Sat Feb 10 01:22:25 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:85 copied) 2024-02-10T01:32:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_6 info 2024-02-10T01:32:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_6 info' 2024-02-10T01:32:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_6 2024-02-10T01:32:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_6': 2024-02-10T01:32:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1165a5908880 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1165a5908880 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:43 2024 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:10 2024 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:43 2024 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:32:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_6 journal status 2024-02-10T01:32:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_6 journal status' 2024-02-10T01:32:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_6 2024-02-10T01:32:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_6 2024-02-10T01:32:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_6 snapshots 2024-02-10T01:32:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_6 snapshots' 2024-02-10T01:32:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_6 2024-02-10T01:32:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 95 snap 128 MiB Sat Feb 10 01:24:07 2024 user 2024-02-10T01:32:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 96 .mirror.non_primary.97797f84-42fa-4648-a6af-57a4b97a77a2.b35bd406-5595-4f6a-bd5b-ac2a636cbbb9 128 MiB Sat Feb 10 01:24:07 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:97 copied) 2024-02-10T01:32:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_7 info 2024-02-10T01:32:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_7 info' 2024-02-10T01:32:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_7 2024-02-10T01:32:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_7': 2024-02-10T01:32:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11aa12959935 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11aa12959935 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:50 2024 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:18 2024 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:50 2024 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:32:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_7 journal status 2024-02-10T01:32:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_7 journal status' 2024-02-10T01:32:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_7 2024-02-10T01:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_7 2024-02-10T01:32:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_7 snapshots' 2024-02-10T01:32:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_7 snapshots 2024-02-10T01:32:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_7 2024-02-10T01:32:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 97 snap 128 MiB Sat Feb 10 01:24:15 2024 user 2024-02-10T01:32:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 98 .mirror.non_primary.c893d42f-80cd-48db-be84-9a6dfee3222c.8625da52-00f8-4254-8f3a-73d6664b8582 128 MiB Sat Feb 10 01:24:16 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:99 copied) 2024-02-10T01:32:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_8 info 2024-02-10T01:32:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_8 info' 2024-02-10T01:32:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_8 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_8': 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1201ba34cb1 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1201ba34cb1 2024-02-10T01:32:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:14:57 2024 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:24 2024 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:14:57 2024 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:32:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_8 journal status 2024-02-10T01:32:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_8 journal status' 2024-02-10T01:32:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_8 2024-02-10T01:32:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_8 2024-02-10T01:32:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_8 snapshots 2024-02-10T01:32:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_8 snapshots' 2024-02-10T01:32:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_8 2024-02-10T01:32:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 99 snap 128 MiB Sat Feb 10 01:24:22 2024 user 2024-02-10T01:32:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 100 .mirror.non_primary.e0049188-c9a3-4f27-bc40-8af6e880240e.ad0f4935-6648-41cc-a51a-7951c23999fc 128 MiB Sat Feb 10 01:24:23 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:101 copied) 2024-02-10T01:32:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-02-10T01:32:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_9 info 2024-02-10T01:32:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_9 info' 2024-02-10T01:32:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_9 2024-02-10T01:32:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image_9': 2024-02-10T01:32:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 128 MiB in 32 objects 2024-02-10T01:32:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-02-10T01:32:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 2 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10adb06ecad1 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10adb06ecad1 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, non-primary 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Feb 10 01:15:08 2024 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Feb 10 01:24:31 2024 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Feb 10 01:15:08 2024 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:32:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: false 2024-02-10T01:32:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_9 journal status 2024-02-10T01:32:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_9 journal status' 2024-02-10T01:32:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_9 2024-02-10T01:32:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: journaling is not enabled for image image_9 2024-02-10T01:32:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image image_9 snapshots 2024-02-10T01:32:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'image image_9 snapshots' 2024-02-10T01:32:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_9 2024-02-10T01:32:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-02-10T01:32:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 101 snap 128 MiB Sat Feb 10 01:24:27 2024 user 2024-02-10T01:32:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 102 .mirror.non_primary.1f78313c-fcfd-4813-9cbf-ce10c8c8e07f.34d7a86c-36ca-40dc-a912-f9c79e2a192a 128 MiB Sat Feb 10 01:24:28 2024 mirror (non-primary peer_uuids:[] 45ca2a06-76be-4782-957c-257ac8a845a6:103 copied) 2024-02-10T01:32:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror rbd_mirroring omap vals 2024-02-10T01:32:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror rbd_mirroring omap vals' 2024-02-10T01:32:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2024-02-10T01:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_042fec75-49d5-40e6-98a0-d964d373c5d4 2024-02-10T01:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 62 38 63 66 37 30 33 33 31 30 |....12b8cf703310| 2024-02-10T01:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_0b89dccc-dd99-4059-94f0-74a8c09ad67b 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 62 62 37 65 66 66 38 32 |....1286bb7eff82| 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_0e8f3781-2b98-408e-b647-688572c90bbb 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 61 33 34 30 38 35 34 64 |....1286a340854d| 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 30 61 64 62 30 36 65 63 61 64 31 |....10adb06ecad1| 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 30 61 64 34 31 37 35 39 35 62 37 |....10ad417595b7| 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_397dba79-e127-495d-94c6-4a204e4c5269 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 61 39 38 66 37 36 38 37 38 34 |....12a98f768784| 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 66 64 39 39 66 36 36 33 |....1165fd99f663| 2024-02-10T01:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 30 61 64 62 37 32 39 62 37 33 63 |....10adb729b73c| 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 32 30 30 62 38 38 33 61 |....11aa200b883a| 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 30 31 32 63 62 36 62 38 30 37 |....12012cb6b807| 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 33 63 62 36 35 66 30 36 |....12863cb65f06| 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 39 37 62 62 31 33 61 64 |....116597bb13ad| 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 61 30 63 66 34 33 35 38 |....1286a0cf4358| 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_67ccb183-b9d9-46d5-8a34-38d09683d20e 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 66 63 61 32 31 64 39 38 65 |....129fca21d98e| 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 65 30 35 37 61 38 61 39 |....1165e057a8a9| 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 30 31 65 62 33 64 65 62 39 31 |....1201eb3deb91| 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 33 31 63 39 34 31 64 35 |....128631c941d5| 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 66 39 33 61 65 36 36 35 36 |....129f93ae6656| 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 34 64 36 62 33 66 66 33 |....11aa4d6b3ff3| 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 33 63 62 30 61 38 38 62 |....12863cb0a88b| 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (15 bytes) : 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0b 00 00 00 31 31 36 35 31 32 36 61 33 64 37 |....1165126a3d7| 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000000f 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 61 35 39 30 38 38 38 30 |....1165a5908880| 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 63 30 38 33 39 39 61 34 |....1286c08399a4| 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_9adfb392-ed84-47b2-9257-a61dc69d11f1 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 35 31 38 37 64 30 65 65 66 |....1295187d0eef| 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 66 63 35 38 37 36 65 66 |....11aafc5876ef| 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 30 31 36 37 36 61 35 61 34 31 |....1201676a5a41| 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_a70dea2d-caa0-45be-8634-f6297695c64e 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 35 66 36 63 66 36 63 65 63 |....1295f6cf6cec| 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_ac3ee523-253b-453f-9e13-a4d28ce2c454 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 66 64 33 66 33 37 39 65 37 |....129fd3f379e7| 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_acaf3e70-c7f2-4137-b839-6ab0448165bb 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 61 39 31 36 36 63 34 64 39 39 |....12a9166c4d99| 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 66 36 33 38 34 61 37 61 |....1165f6384a7a| 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 39 64 38 34 30 33 33 63 |....12869d84033c| 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 66 64 63 31 31 62 32 63 |....11aafdc11b2c| 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 35 63 34 61 64 34 62 33 65 |....1295c4ad4b3e| 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 64 37 63 37 61 65 33 39 |....1165d7c7ae39| 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_c13056e5-b191-4d8c-96f9-917ad3e45ceb 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 63 31 66 61 35 39 65 39 |....1165c1fa59e9| 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 38 31 34 32 65 34 33 35 |....11aa8142e435| 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_c71c4846-b43a-41df-be57-befe30f9b331 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (15 bytes) : 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0b 00 00 00 31 31 36 35 39 64 64 39 66 66 31 |....11659dd9ff1| 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000000f 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 31 32 39 35 39 39 33 35 |....11aa12959935| 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 35 66 39 30 33 66 36 33 66 |....1295f903f63f| 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 30 31 61 37 65 38 39 38 31 62 |....1201a7e8981b| 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 35 36 61 66 30 64 39 32 66 |....12956af0d92f| 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:32:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 38 36 33 63 63 30 31 33 37 62 |....12863cc0137b| 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 30 31 63 34 33 30 32 35 38 62 |....1201c430258b| 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (15 bytes) : 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0b 00 00 00 31 32 30 31 62 61 33 34 63 62 31 |....1201ba34cb1| 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000000f 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 61 39 35 36 37 31 65 37 38 30 |....12a95671e780| 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_e5c1b9f4-c50d-41b1-be2c-c5c92fd22119 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 61 62 61 62 35 30 65 64 |....1165abab50ed| 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_e8b28d26-f2b8-40d9-8c1e-01e760d124f8 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 33 31 65 38 65 64 33 38 |....116531e8ed38| 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 36 35 35 36 63 31 65 66 31 33 |....116556c1ef13| 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 31 61 61 36 38 34 61 30 33 31 61 |....11aa684a031a| 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:global_fa535af5-2334-4635-9fbe-5c4d54ca6819 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (16 bytes) : 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 0c 00 00 00 31 32 39 66 65 64 62 34 32 33 65 38 |....129fedb423e8| 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_10ad417595b7 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 37 38 65 39 66 |..*...$...278e9f| 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 31 65 2d 33 33 65 34 2d 34 36 30 35 2d 39 37 64 |1e-33e4-4605-97d| 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 37 64 37 65 32 38 39 64 65 32 66 64 01 01 |c-7d7e289de2fd..| 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_10adb06ecad1 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 66 37 38 33 31 |..*...$...1f7831| 2024-02-10T01:32:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 33 63 2d 66 63 66 64 2d 34 38 31 33 2d 39 63 62 |3c-fcfd-4813-9cb| 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 66 2d 63 65 31 30 63 38 63 38 65 30 37 66 01 01 |f-ce10c8c8e07f..| 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_10adb729b73c 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 63 65 65 65 36 |..*...$...4ceee6| 2024-02-10T01:32:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 65 36 2d 63 63 36 64 2d 34 61 36 61 2d 38 39 62 |e6-cc6d-4a6a-89b| 2024-02-10T01:32:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 36 2d 32 66 36 31 39 35 33 62 63 31 64 31 01 01 |6-2f61953bc1d1..| 2024-02-10T01:32:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165126a3d7 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 34 37 31 66 63 |..*...$...9471fc| 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 33 30 2d 32 65 38 30 2d 34 32 36 36 2d 39 39 66 |30-2e80-4266-99f| 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 32 2d 36 62 37 64 62 33 30 62 30 38 34 62 01 01 |2-6b7db30b084b..| 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_116531e8ed38 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 38 62 32 38 64 |..*...$...e8b28d| 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 36 2d 66 32 62 38 2d 34 30 64 39 2d 38 63 31 |26-f2b8-40d9-8c1| 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 65 2d 30 31 65 37 36 30 64 31 32 34 66 38 01 01 |e-01e760d124f8..| 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_116556c1ef13 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 61 61 37 38 61 |..*...$...eaa78a| 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 32 2d 37 30 33 37 2d 34 65 32 32 2d 61 31 30 |82-7037-4e22-a10| 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 33 35 33 34 31 66 63 66 36 61 39 39 01 01 |c-35341fcf6a99..| 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_116597bb13ad 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 62 31 36 63 31 |..*...$...5b16c1| 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 63 34 2d 36 37 64 61 2d 34 35 35 62 2d 39 62 61 |c4-67da-455b-9ba| 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 66 2d 64 35 33 31 62 62 30 31 35 65 38 33 01 01 |f-d531bb015e83..| 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11659dd9ff1 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 37 31 63 34 38 |..*...$...c71c48| 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 34 36 2d 62 34 33 61 2d 34 31 64 66 2d 62 65 35 |46-b43a-41df-be5| 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 37 2d 62 65 66 65 33 30 66 39 62 33 33 31 01 01 |7-befe30f9b331..| 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165a5908880 2024-02-10T01:32:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 37 37 39 37 66 |..*...$...97797f| 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 34 2d 34 32 66 61 2d 34 36 34 38 2d 61 36 61 |84-42fa-4648-a6a| 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 66 2d 35 37 61 34 62 39 37 61 37 37 61 32 01 01 |f-57a4b97a77a2..| 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165abab50ed 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 35 63 31 62 39 |..*...$...e5c1b9| 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 66 34 2d 63 35 30 64 2d 34 31 62 31 2d 62 65 32 |f4-c50d-41b1-be2| 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 63 35 63 39 32 66 64 32 32 31 31 39 01 01 |c-c5c92fd22119..| 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165c1fa59e9 2024-02-10T01:32:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 31 33 30 35 36 |..*...$...c13056| 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 65 35 2d 62 31 39 31 2d 34 64 38 63 2d 39 36 66 |e5-b191-4d8c-96f| 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 39 2d 39 31 37 61 64 33 65 34 35 63 65 62 01 01 |9-917ad3e45ceb..| 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165d7c7ae39 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 66 32 64 32 66 |..*...$...bf2d2f| 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 62 65 2d 34 33 33 66 2d 34 66 36 62 2d 62 62 63 |be-433f-4f6b-bbc| 2024-02-10T01:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 66 2d 63 32 66 36 66 37 31 34 34 36 33 65 01 01 |f-c2f6f714463e..| 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165e057a8a9 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 39 65 32 39 39 |..*...$...69e299| 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 61 61 2d 35 38 31 35 2d 34 32 30 32 2d 62 32 63 |aa-5815-4202-b2c| 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 32 2d 39 66 64 33 34 61 39 38 32 61 64 39 01 01 |2-9fd34a982ad9..| 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165f6384a7a 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 66 30 34 36 31 |..*...$...af0461| 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 66 61 2d 65 36 63 38 2d 34 65 32 31 2d 39 30 64 |fa-e6c8-4e21-90d| 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 66 2d 64 31 36 34 62 62 38 31 30 65 31 63 01 01 |f-d164bb810e1c..| 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1165fd99f663 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 33 33 37 61 65 |..*...$...4337ae| 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 64 33 2d 64 39 65 63 2d 34 63 65 61 2d 61 39 39 |d3-d9ec-4cea-a99| 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 35 31 39 33 31 39 33 61 35 65 33 37 01 01 |c-5193193a5e37..| 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aa12959935 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 38 39 33 64 34 |..*...$...c893d4| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 66 2d 38 30 63 64 2d 34 38 64 62 2d 62 65 38 |2f-80cd-48db-be8| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 2d 39 61 36 64 66 65 65 33 32 32 32 63 01 01 |4-9a6dfee3222c..| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aa200b883a 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 66 33 36 33 61 |..*...$...4f363a| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 39 37 2d 39 61 32 30 2d 34 39 31 33 2d 61 31 37 |97-9a20-4913-a17| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 2d 34 64 31 66 61 62 38 61 34 32 35 65 01 01 |4-4d1fab8a425e..| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aa4d6b3ff3 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 39 39 66 65 33 |..*...$...899fe3| 2024-02-10T01:32:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 36 2d 33 31 34 35 2d 34 31 31 30 2d 38 64 31 |26-3145-4110-8d1| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 30 2d 34 64 65 33 63 39 33 34 62 64 61 30 01 01 |0-4de3c934bda0..| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aa684a031a 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 30 66 63 37 63 |..*...$...f0fc7c| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 39 32 2d 39 39 38 35 2d 34 36 32 64 2d 38 34 32 |92-9985-462d-842| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 36 2d 31 64 39 33 34 63 31 32 64 61 61 36 01 01 |6-1d934c12daa6..| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aa8142e435 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 34 37 63 34 65 |..*...$...c47c4e| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 63 36 2d 66 34 37 66 2d 34 33 32 62 2d 39 61 30 |c6-f47f-432b-9a0| 2024-02-10T01:32:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 37 2d 37 62 31 35 37 63 32 37 62 63 62 32 01 01 |7-7b157c27bcb2..| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aafc5876ef 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 35 31 63 32 38 |..*...$...a51c28| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 33 64 2d 31 36 64 61 2d 34 65 38 66 2d 61 38 30 |3d-16da-4e8f-a80| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 62 2d 33 33 62 65 36 65 62 31 35 37 32 62 01 01 |b-33be6eb1572b..| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_11aafdc11b2c 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 32 34 36 30 66 |..*...$...b2460f| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 36 36 2d 31 30 62 31 2d 34 36 63 35 2d 38 30 63 |66-10b1-46c5-80c| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 33 38 36 62 37 33 31 35 38 31 31 32 01 01 |c-386b73158112..| 2024-02-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12012cb6b807 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 31 37 33 37 36 |..*...$...517376| 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 35 66 2d 65 64 64 64 2d 34 35 39 39 2d 38 38 33 |5f-eddd-4599-883| 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 62 2d 33 65 33 65 63 32 65 62 30 66 32 63 01 01 |b-3e3ec2eb0f2c..| 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1201676a5a41 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 36 61 32 31 30 |..*...$...a6a210| 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 64 66 2d 33 66 64 35 2d 34 62 36 31 2d 61 61 62 |df-3fd5-4b61-aab| 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 39 2d 38 37 37 63 30 62 31 30 64 63 34 61 01 01 |9-877c0b10dc4a..| 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1201a7e8981b 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 62 36 37 31 36 |..*...$...cb6716| 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 66 36 2d 32 66 36 64 2d 34 35 63 39 2d 62 37 35 |f6-2f6d-45c9-b75| 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 31 2d 63 62 63 34 62 31 61 62 35 65 30 39 01 01 |1-cbc4b1ab5e09..| 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1201ba34cb1 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 30 30 34 39 31 |..*...$...e00491| 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 38 2d 63 39 61 33 2d 34 66 32 37 2d 62 63 34 |88-c9a3-4f27-bc4| 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 30 2d 38 61 66 36 65 38 38 30 32 34 30 65 01 01 |0-8af6e880240e..| 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1201c430258b 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 33 63 30 36 61 |..*...$...d3c06a| 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 34 35 2d 62 34 37 64 2d 34 39 66 37 2d 61 63 63 |45-b47d-49f7-acc| 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 31 66 35 31 38 63 38 37 31 35 34 37 01 01 |c-1f518c871547..| 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1201eb3deb91 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 35 31 34 63 32 |..*...$...7514c2| 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 63 2d 64 66 66 30 2d 34 32 39 32 2d 62 33 36 |2c-dff0-4292-b36| 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 38 2d 38 61 62 61 31 31 64 65 63 36 34 32 01 01 |8-8aba11dec642..| 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_128631c941d5 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 34 62 36 32 36 |..*...$...84b626| 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 36 38 2d 31 36 34 61 2d 34 34 39 38 2d 62 63 36 |68-164a-4498-bc6| 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 31 2d 63 35 64 63 35 30 65 30 39 36 35 66 01 01 |1-c5dc50e0965f..| 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12863cb0a88b 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 33 62 62 63 32 |..*...$...93bbc2| 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 31 39 2d 38 32 31 62 2d 34 33 65 35 2d 61 37 35 |19-821b-43e5-a75| 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 2d 34 37 31 32 37 30 32 63 32 34 37 65 01 01 |4-4712702c247e..| 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12863cb65f06 2024-02-10T01:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 36 35 61 65 62 |..*...$...565aeb| 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 39 2d 39 62 32 30 2d 34 32 38 65 2d 38 62 61 |89-9b20-428e-8ba| 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 30 2d 38 63 38 61 34 30 31 63 30 64 63 30 01 01 |0-8c8a401c0dc0..| 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12863cc0137b 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 32 32 32 36 66 |..*...$...d2226f| 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 33 2d 38 35 37 66 2d 34 35 31 62 2d 38 64 38 |23-857f-451b-8d8| 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 65 2d 64 36 66 62 61 32 66 62 33 63 63 38 01 01 |e-d6fba2fb3cc8..| 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12869d84033c 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 32 33 62 31 61 |..*...$...b23b1a| 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 62 66 2d 39 34 63 34 2d 34 64 38 62 2d 38 34 39 |bf-94c4-4d8b-849| 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 65 2d 66 62 38 66 32 66 34 39 64 31 61 65 01 01 |e-fb8f2f49d1ae..| 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1286a0cf4358 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 62 35 65 34 36 |..*...$...5b5e46| 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 37 63 2d 30 38 31 39 2d 34 37 32 34 2d 62 62 64 |7c-0819-4724-bbd| 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 31 2d 31 35 37 30 32 61 35 63 38 34 65 66 01 01 |1-15702a5c84ef..| 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1286a340854d 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 65 38 66 33 37 |..*...$...0e8f37| 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 31 2d 32 62 39 38 2d 34 30 38 65 2d 62 36 34 |81-2b98-408e-b64| 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 37 2d 36 38 38 35 37 32 63 39 30 62 62 62 01 01 |7-688572c90bbb..| 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1286bb7eff82 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 62 38 39 64 63 |..*...$...0b89dc| 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 63 63 2d 64 64 39 39 2d 34 30 35 39 2d 39 34 66 |cc-dd99-4059-94f| 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 30 2d 37 34 61 38 63 30 39 61 64 36 37 62 01 01 |0-74a8c09ad67b..| 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1286c08399a4 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 38 39 31 33 64 |..*...$...98913d| 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 65 63 2d 30 66 31 38 2d 34 63 38 31 2d 62 34 34 |ec-0f18-4c81-b44| 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 39 2d 64 38 35 37 32 64 36 38 63 66 65 39 01 01 |9-d8572d68cfe9..| 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1295187d0eef 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 61 64 66 62 33 |..*...$...9adfb3| 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 39 32 2d 65 64 38 34 2d 34 37 62 32 2d 39 32 35 |92-ed84-47b2-925| 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 37 2d 61 36 31 64 63 36 39 64 31 31 66 31 01 01 |7-a61dc69d11f1..| 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12956af0d92f 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 66 63 62 66 63 |..*...$...cfcbfc| 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 66 66 2d 33 37 36 30 2d 34 64 32 62 2d 39 37 37 |ff-3760-4d2b-977| 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 63 2d 33 34 61 35 35 62 62 36 37 62 32 36 01 01 |c-34a55bb67b26..| 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1295c4ad4b3e 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 37 37 62 61 39 |..*...$...b77ba9| 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 36 35 2d 66 35 32 66 2d 34 63 64 32 2d 38 30 31 |65-f52f-4cd2-801| 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 2d 34 63 34 33 31 65 64 38 34 63 39 33 01 01 |4-4c431ed84c93..| 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1295f6cf6cec 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 37 30 64 65 61 |..*...$...a70dea| 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 64 2d 63 61 61 30 2d 34 35 62 65 2d 38 36 33 |2d-caa0-45be-863| 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 2d 66 36 32 39 37 36 39 35 63 36 34 65 01 01 |4-f6297695c64e..| 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_1295f903f63f 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 39 64 38 61 31 |..*...$...c9d8a1| 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 35 33 2d 64 38 34 62 2d 34 34 36 30 2d 62 33 62 |53-d84b-4460-b3b| 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 30 2d 38 30 36 39 64 61 66 66 62 36 39 66 01 01 |0-8069daffb69f..| 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_129f93ae6656 2024-02-10T01:32:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 35 36 39 66 31 |..*...$...8569f1| 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 34 37 2d 31 61 39 39 2d 34 34 38 63 2d 62 63 35 |47-1a99-448c-bc5| 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 38 2d 64 32 33 63 62 64 61 39 65 66 31 38 01 01 |8-d23cbda9ef18..| 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_129fca21d98e 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 37 63 63 62 31 |..*...$...67ccb1| 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 33 2d 62 39 64 39 2d 34 36 64 35 2d 38 61 33 |83-b9d9-46d5-8a3| 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 2d 33 38 64 30 39 36 38 33 64 32 30 65 01 01 |4-38d09683d20e..| 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_129fd3f379e7 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 63 33 65 65 35 |..*...$...ac3ee5| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 32 33 2d 32 35 33 62 2d 34 35 33 66 2d 39 65 31 |23-253b-453f-9e1| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 33 2d 61 34 64 32 38 63 65 32 63 34 35 34 01 01 |3-a4d28ce2c454..| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_129fedb423e8 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 61 35 33 35 61 |..*...$...fa535a| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 66 35 2d 32 33 33 34 2d 34 36 33 35 2d 39 66 62 |f5-2334-4635-9fb| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 65 2d 35 63 34 64 35 34 63 61 36 38 31 39 01 01 |e-5c4d54ca6819..| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12a9166c4d99 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 63 61 66 33 65 |..*...$...acaf3e| 2024-02-10T01:32:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 37 30 2d 63 37 66 32 2d 34 31 33 37 2d 62 38 33 |70-c7f2-4137-b83| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 39 2d 36 61 62 30 34 34 38 31 36 35 62 62 01 01 |9-6ab0448165bb..| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12a95671e780 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 31 65 34 32 66 |..*...$...e1e42f| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 39 33 2d 33 31 65 66 2d 34 30 35 66 2d 62 63 62 |93-31ef-405f-bcb| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 65 2d 66 62 33 65 64 36 37 64 39 66 63 62 01 01 |e-fb3ed67d9fcb..| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12a98f768784 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 39 37 64 62 61 |..*...$...397dba| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 37 39 2d 65 31 32 37 2d 34 39 35 64 2d 39 34 63 |79-e127-495d-94c| 2024-02-10T01:32:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 36 2d 34 61 32 30 34 65 34 63 35 32 36 39 01 01 |6-4a204e4c5269..| 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_12b8cf703310 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (48 bytes) : 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 34 32 66 65 63 |..*...$...042fec| 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 37 35 2d 34 39 64 35 2d 34 30 65 36 2d 39 38 61 |75-49d5-40e6-98a| 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 30 2d 64 39 36 34 64 33 37 33 63 35 64 34 01 01 |0-d964d373c5d4..| 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_mode 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (4 bytes) : 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 00 00 00 |....| 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000004 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_peer_2dc2868b-b7dc-4646-98ea-74d206deca06 2024-02-10T01:32:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (132 bytes) : 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 32 64 63 32 38 36 |..~...$...2dc286| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 38 62 2d 62 37 64 63 2d 34 36 34 36 2d 39 38 65 |8b-b7dc-4646-98e| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 61 2d 37 34 64 32 30 36 64 65 63 61 30 36 08 00 |a-74d206deca06..| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ff ff ff 02 24 00 00 00 34 35 63 61 32 61 30 36 |....$...45ca2a06| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 2d 37 36 62 65 2d 34 37 38 32 2d 39 35 37 63 2d |-76be-4782-957c-| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 32 35 37 61 63 38 61 38 34 35 61 36 32 d2 c6 65 |257ac8a845a62..e| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 f5 96 d1 25 |...%| 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000084 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_uuid 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 32 31 63 39 30 33 66 61 2d 34 39 32 38 2d 34 37 |21c903fa-4928-47| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 33 34 2d 39 36 32 61 2d 31 65 66 36 38 66 63 64 |34-962a-1ef68fcd| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 36 39 36 38 |6968| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_042fec75-49d5-40e6-98a0-d964d373c5d4_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 7a 67 18 25 00 |zg.%.| 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_0b89dccc-dd99-4059-94f0-74a8c09ad67b_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 f0 47 1a 25 00 |.G.%.| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_0e8f3781-2b98-408e-b647-688572c90bbb_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ae 1f 5f 33 00 |.._3.| 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_1f78313c-fcfd-4813-9cbf-ce10c8c8e07f_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ca 3e 60 33 00 |.>`3.| 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_278e9f1e-33e4-4605-97dc-7d7e289de2fd_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 a1 78 e7 1b 00 |.x...| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_397dba79-e127-495d-94c6-4a204e4c5269_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 99 a3 1b 25 00 |...%.| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_4337aed3-d9ec-4cea-a99c-5193193a5e37_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 d6 01 61 33 00 |..a3.| 2024-02-10T01:32:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 1c ab f4 32 00 |...2.| 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_4f363a97-9a20-4913-a174-4d1fab8a425e_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 eb bd 61 33 00 |..a3.| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_5173765f-eddd-4599-883b-3e3ec2eb0f2c_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 3a 79 62 33 00 |:yb3.| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_565aeb89-9b20-428e-8ba0-8c8a401c0dc0_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 93 1e 63 33 00 |..c3.| 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_5b16c1c4-67da-455b-9baf-d531bb015e83_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 2b c6 63 33 00 |+.c3.| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_5b5e467c-0819-4724-bbd1-15702a5c84ef_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 48 5e e8 1b 00 |H^...| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_67ccb183-b9d9-46d5-8a34-38d09683d20e_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 67 ce f5 32 00 |g..2.| 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_69e299aa-5815-4202-b2c2-9fd34a982ad9_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 83 d4 1c 25 00 |...%.| 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_7514c22c-dff0-4292-b368-8aba11dec642_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 f0 63 64 33 00 |.cd3.| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_84b62668-164a-4498-bc61-c5dc50e0965f_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 04 10 65 33 00 |..e3.| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_8569f147-1a99-448c-bc58-d23cbda9ef18_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 f7 02 e9 1b 00 |.....| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_899fe326-3145-4110-8d10-4de3c934bda0_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 92 1f 1e 25 00 |...%.| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_93bbc219-821b-43e5-a754-4712702c247e_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 4d 74 f6 32 00 |Mt.2.| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_9471fc30-2e80-4266-99f2-6b7db30b084b_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 18 9b e9 1b 00 |.....| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_97797f84-42fa-4648-a6af-57a4b97a77a2_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 80 9d 1f 25 00 |...%.| 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_98913dec-0f18-4c81-b449-d8572d68cfe9_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 d4 2a f7 32 00 |.*.2.| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_9adfb392-ed84-47b2-9257-a61dc69d11f1_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 47 db f7 32 00 |G..2.| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_a51c283d-16da-4e8f-a80b-33be6eb1572b_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 c6 2f ea 1b 00 |./...| 2024-02-10T01:32:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_a6a210df-3fd5-4b61-aab9-877c0b10dc4a_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 93 e2 20 25 00 |.. %.| 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_a70dea2d-caa0-45be-8634-f6297695c64e_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 30 7d f8 32 00 |0}.2.| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_ac3ee523-253b-453f-9e13-a4d28ce2c454_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 8b 34 f9 32 00 |.4.2.| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_acaf3e70-c7f2-4137-b839-6ab0448165bb_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 2f bb 65 33 00 |/.e3.| 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_af0461fa-e6c8-4e21-90df-d164bb810e1c_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 fe e0 f9 32 00 |...2.| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ed c3 ea 1b 00 |.....| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_b2460f66-10b1-46c5-80cc-386b73158112_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 86 56 eb 1b 00 |.V...| 2024-02-10T01:32:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_b77ba965-f52f-4cd2-8014-4c431ed84c93_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 45 18 22 25 00 |E."%.| 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 12 d7 fa 32 00 |...2.| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_c13056e5-b191-4d8c-96f9-917ad3e45ceb_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 d3 e9 eb 1b 00 |.....| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_c47c4ec6-f47f-432b-9a07-7b157c27bcb2_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 d5 75 ec 1b 00 |.u...| 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_c71c4846-b43a-41df-be57-befe30f9b331_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 2e 91 fb 32 00 |...2.| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_c893d42f-80cd-48db-be84-9a6dfee3222c_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 4f 76 66 33 00 |Ovf3.| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_c9d8a153-d84b-4460-b3b0-8069daffb69f_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 0c 13 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 aa 6e 50 27 10 00 |...........nP'..| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 23 d2 c6 65 |rting replay#..e| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 f1 21 67 33 00 |.!g3.| 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 f4 40 fc 32 00 |.@.2.| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_cfcbfcff-3760-4d2b-977c-34a55bb67b26_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 4c 14 00 00 00 00 00 00 01 |..,....L........| 2024-02-10T01:32:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 1f 88 47 ee 10 00 |............G...| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 18 d2 c6 65 |rting replay...e| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 c5 e7 fc 32 00 |...2.| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_d2226f23-857f-451b-8d8e-d6fba2fb3cc8_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 b6 56 23 25 00 |.V#%.| 2024-02-10T01:32:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_d3c06a45-b47d-49f7-accc-1f518c871547_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 b4 8f 24 25 00 |..$%.| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_e0049188-c9a3-4f27-bc40-8af6e880240e_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 65 cf 25 25 00 |e.%%.| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ca 13 27 25 00 |..'%.| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_e5c1b9f4-c50d-41b1-be2c-c5c92fd22119_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 36 04 ed 1b 00 |6....| 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_e8b28d26-f2b8-40d9-8c1e-01e760d124f8_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 97 92 ed 1b 00 |.....| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_eaa78a82-7037-4e22-a10c-35341fcf6a99_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 7c 20 ee 1b 00 || ...| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_f0fc7c92-9985-462d-8426-1d934c12daa6_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 bf 10 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 8f c4 23 aa 10 00 |............#...| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 32 d2 c6 65 |rting replay2..e| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ab de ee 1b 00 |.....| 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:remote_status_global_fa535af5-2334-4635-9fbe-5c4d54ca6819_45ca2a06-76be-4782-957c-257ac8a845a6 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e4 11 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 7e 65 47 ee 10 00 |..........~eG...| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 22 d2 c6 65 |rting replay"..e| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 86 65 28 25 00 |.e(%.| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_042fec75-49d5-40e6-98a0-d964d373c5d4 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 34 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |28142,"remote_sn| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 34 32 2c 22 72 65 |":1707528142,"re| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 72 24 9a 27 00 |e"}3..er$.'.| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_0b89dccc-dd99-4059-94f0-74a8c09ad67b 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 30 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |28100,"remote_sn| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 30 30 2c 22 72 65 |":1707528100,"re| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 31 05 9b 27 00 |e"}3..e1..'.| 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_0e8f3781-2b98-408e-b647-688572c90bbb 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 33 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |28035,"remote_sn| 2024-02-10T01:32:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 33 35 2c 22 72 65 |":1707528035,"re| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 55 a9 9b 27 00 |e"}3..eU..'.| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 32 36 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |28267,"remote_sn| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 32 36 37 2c 22 72 65 |":1707528267,"re| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 59 3c 9c 27 00 |e"}3..eY<.'.| 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:32:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 32 33 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |28239,"remote_sn| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 32 33 39 2c 22 72 65 |":1707528239,"re| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 1a a4 d1 24 00 |e"}...e...$.| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_397dba79-e127-495d-94c6-4a204e4c5269 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 32 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |28120,"remote_sn| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 32 30 2c 22 72 65 |":1707528120,"re| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 18 d2 c6 65 60 20 67 2c 00 |e"}...e` g,.| 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 38 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |28381,"remote_sn| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 38 31 2c 22 72 65 |":1707528381,"re| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 22 d2 c6 65 89 04 1a 26 00 |e"}"..e...&.| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (85 bytes) : 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-02-10T01:32:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 33 d2 c6 65 |rting replay3..e| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 4b db 9c 27 00 |K..'.| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000055 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 39 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |28193,"remote_sn| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 39 33 2c 22 72 65 |":1707528193,"re| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 8f 84 d2 24 00 |e"}...e...$.| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 39 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |28199,"remote_sn| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 39 39 2c 22 72 65 |":1707528199,"re| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 59 6b 9d 27 00 |e"}3..eYk.'.| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 34 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |28347,"remote_sn| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 34 37 2c 22 72 65 |":1707528347,"re| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 18 d2 c6 65 1b ff 68 2c 00 |e"}...e..h,.| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 34 30 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |28407,"remote_sn| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 34 30 37 2c 22 72 65 |":1707528407,"re| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 10 f3 9d 27 00 |e"}3..e...'.| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 30 37 32 33 32 30 2e 30 |hot":134072320.0| 2024-02-10T01:32:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 37 32 33 32 30 |bytes":134072320| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 33 30 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |28305,"remote_sn| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 33 30 35 2c 22 72 65 |":1707528305,"re| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 18 d2 c6 65 0b 63 6a 2c 00 |e"}...e.cj,.| 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_67ccb183-b9d9-46d5-8a34-38d09683d20e 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 37 39 39 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |27998,"remote_sn| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 37 39 39 38 2c 22 72 65 |":1707527998,"re| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 22 d2 c6 65 bf 7a 1b 26 00 |e"}"..e.z.&.| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 33 39 34 35 33 34 34 2e 30 |hot":133945344.0| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 39 34 35 33 34 34 |bytes":133945344| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 31 38 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |28186,"remote_sn| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 31 38 36 2c 22 72 65 |":1707528186,"re| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 18 d2 c6 65 13 99 6b 2c 00 |e"}...e..k,.| 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 31 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |28313,"remote_sn| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 31 33 2c 22 72 65 |":1707528313,"re| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 21 89 9e 27 00 |e"}3..e!..'.| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 30 37 39 34 38 38 2e 30 |hot":134079488.0| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 37 39 34 38 38 |bytes":134079488| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 32 39 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |28297,"remote_sn| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 32 39 37 2c 22 72 65 |":1707528297,"re| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 2e d2 c6 65 fd 3e d3 24 00 |e"}...e.>.$.| 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 31 38 35 39 38 34 2e 30 |hot":134185984.0| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 38 35 39 38 34 |bytes":134185984| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 34 34 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |28443,"remote_sn| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 34 34 33 2c 22 72 65 |":1707528443,"re| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 33 d2 c6 65 cd 1b 9f 27 00 |e"}3..e...'.| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 31 38 32 39 31 32 2e 30 |hot":134182912.0| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 38 32 39 31 32 |bytes":134182912| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 32 38 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |28282,"remote_sn| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 32 38 32 2c 22 72 65 |":1707528282,"re| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 2e d2 c6 65 27 d5 d3 24 00 |e"}...e'..$.| 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 32 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |28321,"remote_sn| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 32 31 2c 22 72 65 |":1707528321,"re| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 83 01 a0 27 00 |e"}3..e...'.| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 30 35 31 38 34 30 2e 30 |hot":134051840.0| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 35 31 38 34 30 |bytes":134051840| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 32 37 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |28275,"remote_sn| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 32 37 35 2c 22 72 65 |":1707528275,"re| 2024-02-10T01:32:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 18 d2 c6 65 e2 d3 6c 2c 00 |e"}...e..l,.| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 32 34 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |28246,"remote_sn| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 32 34 36 2c 22 72 65 |":1707528246,"re| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 27 9c a0 27 00 |e"}3..e'..'.| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 37 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |28375,"remote_sn| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 37 35 2c 22 72 65 |":1707528375,"re| 2024-02-10T01:32:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 00 64 d4 24 00 |e"}...e.d.$.| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_9adfb392-ed84-47b2-9257-a61dc69d11f1 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 30 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |28109,"remote_sn| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 30 39 2c 22 72 65 |":1707528109,"re| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 22 d2 c6 65 50 c7 1c 26 00 |e"}"..eP..&.| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 e0 14 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 de 79 a7 c9 10 00 |...........y....| 2024-02-10T01:32:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 33 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |28335,"remote_sn| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 33 35 2c 22 72 65 |":1707528335,"re| 2024-02-10T01:32:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 33 d2 c6 65 e6 2b a1 27 00 |e"}3..e.+.'.| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 33 39 36 36 38 34 38 2e 30 |hot":133966848.0| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 39 36 36 38 34 38 |bytes":133966848| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 33 36 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |28368,"remote_sn| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 33 36 38 2c 22 72 65 |":1707528368,"re| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 22 d2 c6 65 c0 0f 1e 26 00 |e"}"..e...&.| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_a70dea2d-caa0-45be-8634-f6297695c64e 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 34 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |28040,"remote_sn| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 34 30 2c 22 72 65 |":1707528040,"re| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 22 d2 c6 65 54 60 1f 26 00 |e"}"..eT`.&.| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_ac3ee523-253b-453f-9e13-a4d28ce2c454 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 31 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |28012,"remote_sn| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 31 32 2c 22 72 65 |":1707528012,"re| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 18 d2 c6 65 a2 0a 6e 2c 00 |e"}...e..n,.| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_acaf3e70-c7f2-4137-b839-6ab0448165bb 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 38 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |28084,"remote_sn| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 38 34 2c 22 72 65 |":1707528084,"re| 2024-02-10T01:32:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 b8 f5 d4 24 00 |e"}...e...$.| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 31 31 31 32 33 32 2e 30 |hot":134111232.0| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 31 31 32 33 32 |bytes":134111232| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 33 32 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |28328,"remote_sn| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 33 32 38 2c 22 72 65 |":1707528328,"re| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 18 d2 c6 65 08 48 6f 2c 00 |e"}...e.Ho,.| 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 36 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |28064,"remote_sn| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 36 34 2c 22 72 65 |":1707528064,"re| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 18 d2 c6 65 5f 9c 70 2c 00 |e"}...e_.p,.| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 33 33 36 39 38 35 36 2e 30 |hot":133369856.0| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 33 36 39 38 35 36 |bytes":133369856| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 33 36 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |28361,"remote_sn| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 33 36 31 2c 22 72 65 |":1707528361,"re| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 22 d2 c6 65 18 ae 20 26 00 |e"}"..e.. &.| 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:32:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (299 bytes) : 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 36 36 39 37 30 31 31 32 2e 30 2c |hot":66970112.0,| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 79 74 65 73 22 3a 31 33 33 39 34 30 32 32 34 2c |ytes":133940224,| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2024-02-10T01:32:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 32 |imestamp":170752| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 38 33 39 39 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |8399,"remote_sna| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 3a 31 37 30 37 35 32 38 33 39 39 2c 22 72 65 70 |:1707528399,"rep| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 22 7d 22 d2 c6 65 f2 e6 21 26 00 |"}"..e..!&.| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012b 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 33 35 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |28354,"remote_sn| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 33 35 34 2c 22 72 65 |":1707528354,"re| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 18 d2 c6 65 be cc 71 2c 00 |e"}...e..q,.| 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_c13056e5-b191-4d8c-96f9-917ad3e45ceb 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 31 33 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |28130,"remote_sn| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 31 33 30 2c 22 72 65 |":1707528130,"re| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 b1 88 d5 24 00 |e"}...e...$.| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 33 37 34 38 37 33 36 2e 30 |hot":133748736.0| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 34 38 37 33 36 |bytes":133748736| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 34 31 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |28415,"remote_sn| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 34 31 35 2c 22 72 65 |":1707528415,"re| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 2e d2 c6 65 43 13 d6 24 00 |e"}...eC..$.| 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_c71c4846-b43a-41df-be57-befe30f9b331 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 32 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |28023,"remote_sn| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 32 33 2c 22 72 65 |":1707528023,"re| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 65 9e d6 24 00 |e"}...ee..$.| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 32 39 30 34 39 36 30 2e 30 |hot":132904960.0| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 32 39 30 34 39 36 30 |bytes":132904960| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 32 35 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |28254,"remote_sn| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 32 35 34 2c 22 72 65 |":1707528254,"re| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 18 d2 c6 65 64 11 73 2c 00 |e"}...ed.s,.| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (299 bytes) : 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 36 37 30 33 37 36 39 36 2e 30 2c |hot":67037696.0,| 2024-02-10T01:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 37 35 33 39 32 2c |ytes":134075392,| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 32 |imestamp":170752| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 38 33 39 33 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |8393,"remote_sna| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 3a 31 37 30 37 35 32 38 33 39 33 2c 22 72 65 70 |:1707528393,"rep| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 22 7d 22 d2 c6 65 2a 58 23 26 00 |"}"..e*X#&.| 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012b 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 33 33 38 30 30 39 36 2e 30 |hot":133380096.0| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 33 38 30 30 39 36 |bytes":133380096| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 33 34 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |28341,"remote_sn| 2024-02-10T01:32:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 33 34 31 2c 22 72 65 |":1707528341,"re| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 22 d2 c6 65 ed a3 24 26 00 |e"}"..e..$&.| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (299 bytes) : 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 36 36 33 38 35 34 30 38 2e 30 2c |hot":66385408.0,| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 79 74 65 73 22 3a 31 33 32 37 37 30 38 31 36 2c |ytes":132770816,| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 32 |imestamp":170752| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 38 34 33 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |8430,"remote_sna| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 3a 31 37 30 37 35 32 38 34 33 30 2c 22 72 65 70 |:1707528430,"rep| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 22 7d 22 d2 c6 65 9f ed 25 26 00 |"}"..e..%&.| 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012b 2024-02-10T01:32:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (299 bytes) : 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 36 36 39 30 34 30 36 34 2e 30 2c |hot":66904064.0,| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 79 74 65 73 22 3a 31 33 33 38 30 38 31 32 38 2c |ytes":133808128,| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2024-02-10T01:32:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 32 |imestamp":170752| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 38 34 32 33 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |8423,"remote_sna| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 3a 31 37 30 37 35 32 38 34 32 33 2c 22 72 65 70 |:1707528423,"rep| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 22 7d 18 d2 c6 65 52 7c 74 2c 00 |"}...eR|t,.| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012b 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 31 30 35 30 38 38 2e 30 |hot":134105088.0| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 30 35 30 38 38 |bytes":134105088| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 32 38 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |28289,"remote_sn| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 32 38 39 2c 22 72 65 |":1707528289,"re| 2024-02-10T01:32:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 18 d2 c6 65 07 c4 75 2c 00 |e"}...e..u,.| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 34 31 30 35 30 38 38 2e 30 |hot":134105088.0| 2024-02-10T01:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 30 35 30 38 38 |bytes":134105088| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 32 36 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |28261,"remote_sn| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 32 36 31 2c 22 72 65 |":1707528261,"re| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 22 d2 c6 65 0f 23 27 26 00 |e"}"..e.#'&.| 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb 2024-02-10T01:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 37 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |28076,"remote_sn| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 37 36 2c 22 72 65 |":1707528076,"re| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 2c 2d d7 24 00 |e"}...e,-.$.| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_e5c1b9f4-c50d-41b1-be2c-c5c92fd22119 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 39 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |28091,"remote_sn| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 39 31 2c 22 72 65 |":1707528091,"re| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 02 bc d7 24 00 |e"}...e...$.| 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_e8b28d26-f2b8-40d9-8c1e-01e760d124f8 2024-02-10T01:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 35 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |28054,"remote_sn| 2024-02-10T01:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 35 34 2c 22 72 65 |":1707528054,"re| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 2e d2 c6 65 b6 46 d8 24 00 |e"}...e.F.$.| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (299 bytes) : 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 65 11 00 00 00 00 00 00 01 |..,....e........| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c2 07 e0 32 10 00 |.............2..| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 36 36 39 31 36 33 35 32 2e 30 2c |hot":66916352.0,| 2024-02-10T01:32:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 79 74 65 73 22 3a 31 33 33 38 33 32 37 30 34 2c |ytes":133832704,| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 32 |imestamp":170752| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 38 34 33 36 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |8436,"remote_sna| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 3a 31 37 30 37 35 32 38 34 33 36 2c 22 72 65 70 |:1707528436,"rep| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 22 7d 2e d2 c6 65 fb ce d8 24 00 |"}...e...$.| 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012b 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:32:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (300 bytes) : 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 95 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ea e8 bf 80 10 00 |................| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 31 33 33 37 34 31 35 36 38 2e 30 |hot":133741568.0| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 34 31 35 36 38 |bytes":133741568| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 32 38 33 38 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |28387,"remote_sn| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 22 3a 31 37 30 37 35 32 38 33 38 37 2c 22 72 65 |":1707528387,"re| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000120 65 22 7d 22 d2 c6 65 71 60 28 26 00 |e"}"..eq`(&.| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000012c 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:status_global_fa535af5-2334-4635-9fbe-5c4d54ca6819 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (284 bytes) : 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 2c 00 00 00 08 86 12 00 00 00 00 00 00 01 |..,.............| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 f9 3d ad e8 10 00 |...........=....| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 02 00 00 00 ac 15 0f 3c 00 00 00 00 00 00 |.........<......| 2024-02-10T01:32:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 37 35 |timestamp":17075| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000d0 32 38 30 34 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |28048,"remote_sn| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:000000f0 22 3a 31 37 30 37 35 32 38 30 34 38 2c 22 72 65 |":1707528048,"re| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000110 65 22 7d 18 d2 c6 65 be fe 76 2c 00 |e"}...e..v,.| 2024-02-10T01:32:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:0000011c 2024-02-10T01:32:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror rbd_mirror_leader omap vals 2024-02-10T01:32:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2024-02-10T01:32:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_042fec75-49d5-40e6-98a0-d964d373c5d4 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 01 d1 |..........5344..| 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 76 1e a7 32 0a 00 00 00 01 01 04 00 00 00 |.ev..2..........| 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_0b89dccc-dd99-4059-94f0-74a8c09ad67b 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 2e 9f 0a 34 0a 00 00 00 01 01 04 00 00 00 |.e...4..........| 2024-02-10T01:32:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_0e8f3781-2b98-408e-b647-688572c90bbb 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 51 bd 0a 34 0a 00 00 00 01 01 04 00 00 00 |.eQ..4..........| 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_1f78313c-fcfd-4813-9cbf-ce10c8c8e07f 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 01 d1 |..........5344..| 2024-02-10T01:32:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 50 36 a7 32 0a 00 00 00 01 01 04 00 00 00 |.eP6.2..........| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_278e9f1e-33e4-4605-97dc-7d7e289de2fd 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 01 d1 |..........4453..| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 12 43 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e.C.2..........| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_397dba79-e127-495d-94c6-4a204e4c5269 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 01 d1 |..........4742..| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 90 50 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e.P.2..........| 2024-02-10T01:32:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_4337aed3-d9ec-4cea-a99c-5193193a5e37 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 01 d1 |..........4757..| 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 2e 5c a7 32 0a 00 00 00 01 01 04 00 00 00 |.e.\.2..........| 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_4ceee6e6-cc6d-4a6a-89b6-2f61953bc1d1 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 01 d1 |..........5344..| 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 5a 67 a7 32 0a 00 00 00 01 01 04 00 00 00 |.eZg.2..........| 2024-02-10T01:32:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_4f363a97-9a20-4913-a174-4d1fab8a425e 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 01 d1 |..........4453..| 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 ca 74 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e.t.2..........| 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_5173765f-eddd-4599-883b-3e3ec2eb0f2c 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 f9 d5 0a 34 0a 00 00 00 01 01 04 00 00 00 |.e...4..........| 2024-02-10T01:32:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_565aeb89-9b20-428e-8ba0-8c8a401c0dc0 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 01 d1 |..........4742..| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 3b 81 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e;..2..........| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_5b16c1c4-67da-455b-9baf-d531bb015e83 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 56 f7 0a 34 0a 00 00 00 01 01 04 00 00 00 |.eV..4..........| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_5b5e467c-0819-4724-bbd1-15702a5c84ef 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 df cf |..........4742..| 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 32 16 10 2c 0a 00 00 00 01 01 04 00 00 00 |.e2..,..........| 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_67ccb183-b9d9-46d5-8a34-38d09683d20e 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 01 d1 |..........4757..| 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 1e 94 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e...2..........| 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_69e299aa-5815-4202-b2c2-9fd34a982ad9 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 df cf |..........4742..| 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 85 53 10 2c 0a 00 00 00 01 01 04 00 00 00 |.e.S.,..........| 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_7514c22c-dff0-4292-b368-8aba11dec642 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 fe 10 0b 34 0a 00 00 00 01 01 04 00 00 00 |.e...4..........| 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_84b62668-164a-4498-bc61-c5dc50e0965f 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 25 d0 |..........4453%.| 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 1a 84 c8 2d 0a 00 00 00 01 01 04 00 00 00 |.e...-..........| 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_8569f147-1a99-448c-bc58-d23cbda9ef18 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 16 26 0b 34 0a 00 00 00 01 01 04 00 00 00 |.e.&.4..........| 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_899fe326-3145-4110-8d10-4de3c934bda0 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 df cf |..........4453..| 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 89 8b 10 2c 0a 00 00 00 01 01 04 00 00 00 |.e...,..........| 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_93bbc219-821b-43e5-a754-4712702c247e 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 01 d1 |..........5344..| 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 a3 a2 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e...2..........| 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_9471fc30-2e80-4266-99f2-6b7db30b084b 2024-02-10T01:32:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 df cf |..........4742..| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 c7 a7 10 2c 0a 00 00 00 01 01 04 00 00 00 |.e...,..........| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_97797f84-42fa-4648-a6af-57a4b97a77a2 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 69 37 0b 34 0a 00 00 00 01 01 04 00 00 00 |.ei7.4..........| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_98913dec-0f18-4c81-b449-d8572d68cfe9 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 01 d1 |..........4453..| 2024-02-10T01:32:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 fe b0 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e...2..........| 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_9adfb392-ed84-47b2-9257-a61dc69d11f1 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 ae cf |..........4757..| 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 b2 12 a5 34 0a 00 00 00 01 01 04 00 00 00 |.e...4..........| 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_a51c283d-16da-4e8f-a80b-33be6eb1572b 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 33 34 34 ed d0 |..........5344..| 2024-02-10T01:32:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 85 4e 0b 34 0a 00 00 00 01 01 04 00 00 00 |.e.N.4..........| 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_a6a210df-3fd5-4b61-aab9-877c0b10dc4a 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 f4 ce |..........4757..| 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 bf 03 27 26 0a 00 00 00 01 01 04 00 00 00 |.e..'&..........| 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_a70dea2d-caa0-45be-8634-f6297695c64e 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 69 cf |..........4757i.| 2024-02-10T01:32:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 dc 28 70 33 0a 00 00 00 01 01 04 00 00 00 |.e.(p3..........| 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_ac3ee523-253b-453f-9e13-a4d28ce2c454 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 99 cf |..........4742..| 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 74 ad 5d 2a 0a 00 00 00 01 01 04 00 00 00 |.et.]*..........| 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_acaf3e70-c7f2-4137-b839-6ab0448165bb 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 25 d0 |..........4453%.| 2024-02-10T01:32:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 da b2 c8 2d 0a 00 00 00 01 01 04 00 00 00 |.e...-..........| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_af0461fa-e6c8-4e21-90df-d164bb810e1c 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 25 d0 |..........4742%.| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 8f c6 c8 2d 0a 00 00 00 01 01 04 00 00 00 |.e...-..........| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_b23b1abf-94c4-4d8b-849e-fb8f2f49d1ae 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 81 cf |..........4742..| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 0a 06 ea 0e 0a 00 00 00 01 01 04 00 00 00 |.e..............| 2024-02-10T01:32:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_b2460f66-10b1-46c5-80cc-386b73158112 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 99 cf |..........4757..| 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 e8 c1 5d 2a 0a 00 00 00 01 01 04 00 00 00 |.e..]*..........| 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_b77ba965-f52f-4cd2-8014-4c431ed84c93 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 e9 ce |..........4757..| 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 9d c8 56 25 0a 00 00 00 01 01 04 00 00 00 |.e..V%..........| 2024-02-10T01:32:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_bf2d2fbe-433f-4f6b-bbcf-c2f6f714463e 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 01 d1 |..........4742..| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 09 c3 a7 32 0a 00 00 00 01 01 04 00 00 00 |.e...2..........| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_c13056e5-b191-4d8c-96f9-917ad3e45ceb 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 c3 cf |..........4453..| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 7d 82 d4 34 0a 00 00 00 01 01 04 00 00 00 |.e}..4..........| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_c47c4ec6-f47f-432b-9a07-7b157c27bcb2 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 48 cf |..........4453H.| 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 1b ed b4 29 0a 00 00 00 01 01 04 00 00 00 |.e...)..........| 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_c71c4846-b43a-41df-be57-befe30f9b331 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 58 cf |..........4453X.| 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 13 0f f2 34 0a 00 00 00 01 01 04 00 00 00 |.e...4..........| 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_c893d42f-80cd-48db-be84-9a6dfee3222c 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 48 cf |..........4742H.| 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 3e ff b4 29 0a 00 00 00 01 01 04 00 00 00 |.e>..)..........| 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_c9d8a153-d84b-4460-b3b0-8069daffb69f 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 f4 ce |..........4757..| 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 f0 15 27 26 0a 00 00 00 01 01 04 00 00 00 |.e..'&..........| 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_cb6716f6-2f6d-45c9-b751-cbc4b1ab5e09 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 f4 ce |..........4757..| 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 aa 22 27 26 0a 00 00 00 01 01 04 00 00 00 |.e."'&..........| 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_cfcbfcff-3760-4d2b-977c-34a55bb67b26 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 1e cf |..........4757..| 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 90 2e bf 0d 0a 00 00 00 01 01 04 00 00 00 |.e..............| 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_d2226f23-857f-451b-8d8e-d6fba2fb3cc8 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 10 cf |..........4742..| 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 84 a6 ef 31 0a 00 00 00 01 01 04 00 00 00 |.e...1..........| 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_d3c06a45-b47d-49f7-accc-1f518c871547 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 f4 ce |..........4742..| 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 1d 2f 27 26 0a 00 00 00 01 01 04 00 00 00 |.e./'&..........| 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_e0049188-c9a3-4f27-bc40-8af6e880240e 2024-02-10T01:32:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 f4 ce |..........4757..| 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 03 3d 27 26 0a 00 00 00 01 01 04 00 00 00 |.e.='&..........| 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_e1e42f93-31ef-405f-bcbe-fb3ed67d9fcb 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 df cf |..........4453..| 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 38 ee 10 2c 0a 00 00 00 01 01 04 00 00 00 |.e8..,..........| 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_e5c1b9f4-c50d-41b1-be2c-c5c92fd22119 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 9c cf |..........4453..| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 3a b6 79 34 0a 00 00 00 01 01 04 00 00 00 |.e:.y4..........| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_e8b28d26-f2b8-40d9-8c1e-01e760d124f8 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 77 cf |..........4453w.| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 40 2a 98 0e 0a 00 00 00 01 01 04 00 00 00 |.e@*............| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_eaa78a82-7037-4e22-a10c-35341fcf6a99 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 34 35 33 25 cf |..........4453%.| 2024-02-10T01:32:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 7b 0c 5c 32 0a 00 00 00 01 01 04 00 00 00 |.e{.\2..........| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_f0fc7c92-9985-462d-8426-1d934c12daa6 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 35 37 48 cf |..........4757H.| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 6d 11 b5 29 0a 00 00 00 01 01 04 00 00 00 |.em..)..........| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image_map_fa535af5-2334-4635-9fbe-5c4d54ca6819 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 32 df cf |..........4742..| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 c6 65 13 07 11 2c 0a 00 00 00 01 01 04 00 00 00 |.e...,..........| 2024-02-10T01:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 00 00 00 00 |....| 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4453 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4742 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4757 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_5344 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:32:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror ns1 images 2024-02-10T01:32:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror ns1 images' 2024-02-10T01:32:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2024-02-10T01:32:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirrorns1 mirror pool info 2024-02-10T01:32:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirrorns1 mirror pool info' 2024-02-10T01:32:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2024-02-10T01:32:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mode: image 2024-02-10T01:32:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirrorns1 mirror pool status 2024-02-10T01:32:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirrorns1 mirror pool status' 2024-02-10T01:32:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:health: OK 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:daemon health: OK 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image health: OK 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:images: 0 total 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:DAEMONS 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4450: 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4453 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.2 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: true 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4739: 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4742 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.3 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4754: 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4757 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.1 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 5341: 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 5344 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.0 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:IMAGES 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2024-02-10T01:32:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2024-02-10T01:32:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2024-02-10T01:32:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-02-10T01:32:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_mode 2024-02-10T01:32:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (4 bytes) : 2024-02-10T01:32:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 00 00 00 |....| 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000004 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_uuid 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 64 61 65 64 39 64 37 65 2d 62 38 66 61 2d 34 34 |daed9d7e-b8fa-44| 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 61 64 2d 61 30 30 39 2d 32 35 33 66 36 33 38 63 |ad-a009-253f638c| 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 34 38 32 31 |4821| 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2024-02-10T01:32:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2024-02-10T01:32:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4453 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4742 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4757 2024-02-10T01:32:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:32:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror ns2 images' 2024-02-10T01:32:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror ns2 images 2024-02-10T01:32:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2024-02-10T01:32:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirrorns2 mirror pool info 2024-02-10T01:32:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirrorns2 mirror pool info' 2024-02-10T01:32:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2024-02-10T01:32:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mode: image 2024-02-10T01:32:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirrorns2 mirror pool status' 2024-02-10T01:32:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirrorns2 mirror pool status 2024-02-10T01:32:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2024-02-10T01:32:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:health: OK 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:daemon health: OK 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image health: OK 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:images: 0 total 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:DAEMONS 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4450: 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4453 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.2 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: true 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4739: 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4742 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.3 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4754: 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4757 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.1 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 5341: 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 5344 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.0 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:IMAGES 2024-02-10T01:32:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2024-02-10T01:32:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2024-02-10T01:32:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2024-02-10T01:32:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-02-10T01:32:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_mode 2024-02-10T01:32:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (4 bytes) : 2024-02-10T01:32:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 00 00 00 |....| 2024-02-10T01:32:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000004 2024-02-10T01:32:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_uuid 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 30 62 33 66 35 39 30 32 2d 38 34 39 30 2d 34 66 |0b3f5902-8490-4f| 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 31 61 2d 38 62 65 63 2d 66 39 38 34 37 63 65 30 |1a-8bec-f9847ce0| 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 39 31 66 65 |91fe| 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2024-02-10T01:32:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2024-02-10T01:32:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-02-10T01:32:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4453 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4742 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4757 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-02-10T01:32:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:32:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent images 2024-02-10T01:32:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent images' 2024-02-10T01:32:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2024-02-10T01:32:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent mirror pool info 2024-02-10T01:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent mirror pool info' 2024-02-10T01:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2024-02-10T01:32:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mode: image 2024-02-10T01:32:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Site Name: cluster1 2024-02-10T01:32:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Peer Sites: 2024-02-10T01:32:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:UUID: 2b77c1ab-96e8-4c99-b228-f5be0d8b4938 2024-02-10T01:32:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Name: cluster2 2024-02-10T01:32:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirror UUID: 0a91117b-5ec2-41da-a8a7-64a0d771311c 2024-02-10T01:32:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Direction: rx-tx 2024-02-10T01:32:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Client: client.mirror 2024-02-10T01:32:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent mirror pool status 2024-02-10T01:32:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent mirror pool status' 2024-02-10T01:32:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:health: OK 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:daemon health: OK 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:image health: OK 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:images: 0 total 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:DAEMONS 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4450: 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4455 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.2 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: true 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4739: 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4745 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.3 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 4754: 2024-02-10T01:32:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 4760 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.1 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:service 5341: 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: instance_id: 5347 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: client_id: mirror.0 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: hostname: smithi060 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: version: 19.0.0-1361-g290212a1 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: leader: false 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: health: OK 2024-02-10T01:32:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:IMAGES 2024-02-10T01:32:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2024-02-10T01:32:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2024-02-10T01:32:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2024-02-10T01:32:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_mode 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (4 bytes) : 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 01 00 00 00 |....| 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000004 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_peer_2b77c1ab-96e8-4c99-b228-f5be0d8b4938 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (132 bytes) : 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 32 62 37 37 63 31 |..~...$...2b77c1| 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 61 62 2d 39 36 65 38 2d 34 63 39 39 2d 62 32 32 |ab-96e8-4c99-b22| 2024-02-10T01:32:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 38 2d 66 35 62 65 30 64 38 62 34 39 33 38 08 00 |8-f5be0d8b4938..| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000050 ff ff ff 02 24 00 00 00 30 61 39 31 31 31 37 62 |....$...0a91117b| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000060 2d 35 65 63 32 2d 34 31 64 61 2d 61 38 61 37 2d |-5ec2-41da-a8a7-| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000070 36 34 61 30 64 37 37 31 33 31 31 63 32 d2 c6 65 |64a0d771311c2..e| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000080 75 55 36 34 |uU64| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000084 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:mirror_uuid 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (36 bytes) : 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000000 37 30 35 66 62 62 36 63 2d 62 63 32 62 2d 34 63 |705fbb6c-bc2b-4c| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000010 31 64 2d 61 62 61 39 2d 32 36 37 39 30 65 66 65 |1d-aba9-26790efe| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000020 32 31 61 37 |21a7| 2024-02-10T01:32:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:00000024 2024-02-10T01:32:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2024-02-10T01:32:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2024-02-10T01:32:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-02-10T01:32:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4455 2024-02-10T01:32:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4745 2024-02-10T01:32:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:instance_4760 2024-02-10T01:32:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:value (0 bytes) : 2024-02-10T01:32:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:32:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent ns1 images 2024-02-10T01:32:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent ns1 images' 2024-02-10T01:32:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2024-02-10T01:32:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: namespace 'ns1' does not exist. 2024-02-10T01:32:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (2) No such file or directory 2024-02-10T01:32:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parentns1 mirror pool info 2024-02-10T01:32:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool info' 2024-02-10T01:32:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2024-02-10T01:32:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: namespace 'ns1' does not exist. 2024-02-10T01:32:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parentns1 mirror pool status 2024-02-10T01:32:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool status' 2024-02-10T01:32:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-02-10T01:32:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: namespace 'ns1' does not exist. 2024-02-10T01:32:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2024-02-10T01:32:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2024-02-10T01:32:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2024-02-10T01:32:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-02-10T01:32:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2024-02-10T01:32:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-02-10T01:32:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2024-02-10T01:32:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-02-10T01:32:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2024-02-10T01:32:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:32:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent ns2 images 2024-02-10T01:32:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent ns2 images' 2024-02-10T01:32:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2024-02-10T01:32:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: namespace 'ns2' does not exist. 2024-02-10T01:32:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (2) No such file or directory 2024-02-10T01:32:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool info' 2024-02-10T01:32:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parentns2 mirror pool info 2024-02-10T01:32:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2024-02-10T01:32:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: namespace 'ns2' does not exist. 2024-02-10T01:32:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool status' 2024-02-10T01:32:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parentns2 mirror pool status 2024-02-10T01:32:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-02-10T01:32:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: namespace 'ns2' does not exist. 2024-02-10T01:32:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2024-02-10T01:32:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2024-02-10T01:32:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2024-02-10T01:32:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-02-10T01:32:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2024-02-10T01:32:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-02-10T01:32:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2024-02-10T01:32:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-02-10T01:32:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2024-02-10T01:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-02-10T01:32:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 status 2024-02-10T01:32:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 status' 2024-02-10T01:32:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:32:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 -s 2024-02-10T01:37:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T01:37:38.374+0000 7f41fcbd0640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T01:37:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T01:37:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:37:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 service dump 2024-02-10T01:38:51.883 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2024-02-10T01:38:51.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2024-02-10T01:38:51.885 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:*** Caught signal (Aborted) ** 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: in thread 7fb47f018640 thread_name:rbd-mirror 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb47ee54db0] 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7fb47ee9c39a] 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fb47f2d5500] 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 5: /lib64/librados.so.2(+0x143f1b) [0x7fb480c26f1b] 2024-02-10T01:38:51.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f5bcbedc58] 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55f5bcbefad8] 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55f5bcbcedf4] 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 9: main() 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb47ee3feb0] 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 11: __libc_start_main() 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 12: _start() 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:2024-02-10T01:38:51.886+0000 7fb47f018640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: in thread 7fb47f018640 thread_name:rbd-mirror 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2024-02-10T01:38:51.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb47ee54db0] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7fb47ee9c39a] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fb47f2d5500] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 5: /lib64/librados.so.2(+0x143f1b) [0x7fb480c26f1b] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f5bcbedc58] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55f5bcbefad8] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55f5bcbcedf4] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 9: main() 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb47ee3feb0] 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 11: __libc_start_main() 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 12: _start() 2024-02-10T01:38:51.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:38:51.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2024-02-10T01:38:51.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: -1889> 2024-02-10T01:28:42.733+0000 7fb47bb02640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57934) UID: 1000 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 0> 2024-02-10T01:38:51.886+0000 7fb47f018640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: in thread 7fb47f018640 thread_name:rbd-mirror 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb47ee54db0] 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7fb47ee9c39a] 2024-02-10T01:38:51.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fb47f2d5500] 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 5: /lib64/librados.so.2(+0x143f1b) [0x7fb480c26f1b] 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f5bcbedc58] 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55f5bcbefad8] 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55f5bcbcedf4] 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 9: main() 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb47ee3feb0] 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 11: __libc_start_main() 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 12: _start() 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:38:51.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: -7546> 2024-02-10T01:28:42.733+0000 7fb47bb02640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57934) UID: 1000 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: -7545> 2024-02-10T01:38:51.886+0000 7fb47f018640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: in thread 7fb47f018640 thread_name:rbd-mirror 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fb47ee54db0] 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7fb47ee9c39a] 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:38:51.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fb47f2d5500] 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 5: /lib64/librados.so.2(+0x143f1b) [0x7fb480c26f1b] 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f5bcbedc58] 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x55f5bcbefad8] 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x55f5bcbcedf4] 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 9: main() 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fb47ee3feb0] 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 11: __libc_start_main() 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 12: _start() 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:38:51.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr: 2024-02-10T01:38:52.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi060.stderr:daemon-helper: command crashed with signal 6 2024-02-10T01:39:06.692 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.0 2024-02-10T01:39:06.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 6 2024-02-10T01:39:06.692 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-02-10T01:39:06.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:*** Caught signal (Aborted) ** 2024-02-10T01:39:06.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: in thread 7faf54625640 thread_name:rbd-mirror 2024-02-10T01:39:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:39:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7faf54454db0] 2024-02-10T01:39:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7faf5449c39a] 2024-02-10T01:39:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:39:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faf548d5500] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 5: rbd-mirror(+0x36bd2b) [0x55d31872ad2b] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55d31873b2a3] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55d31872e5d0] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 8: rbd-mirror(+0x9896f8) [0x55d318d486f8] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 9: rbd-mirror(+0x9896e0) [0x55d318d486e0] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55d31872caea] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 11: main() 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7faf5443feb0] 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 13: __libc_start_main() 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 14: _start() 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:2024-02-10T01:39:06.692+0000 7faf54625640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: in thread 7faf54625640 thread_name:rbd-mirror 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2024-02-10T01:39:06.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7faf54454db0] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7faf5449c39a] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faf548d5500] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 5: rbd-mirror(+0x36bd2b) [0x55d31872ad2b] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55d31873b2a3] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55d31872e5d0] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 8: rbd-mirror(+0x9896f8) [0x55d318d486f8] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 9: rbd-mirror(+0x9896e0) [0x55d318d486e0] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55d31872caea] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 11: main() 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7faf5443feb0] 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 13: __libc_start_main() 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 14: _start() 2024-02-10T01:39:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:39:06.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2024-02-10T01:39:06.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: -9716> 2024-02-10T01:28:58.548+0000 7faf51066640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59960) UID: 1000 2024-02-10T01:39:06.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 0> 2024-02-10T01:39:06.692+0000 7faf54625640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:39:06.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: in thread 7faf54625640 thread_name:rbd-mirror 2024-02-10T01:39:06.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2024-02-10T01:39:06.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:39:06.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7faf54454db0] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7faf5449c39a] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faf548d5500] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 5: rbd-mirror(+0x36bd2b) [0x55d31872ad2b] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55d31873b2a3] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55d31872e5d0] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 8: rbd-mirror(+0x9896f8) [0x55d318d486f8] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 9: rbd-mirror(+0x9896e0) [0x55d318d486e0] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55d31872caea] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 11: main() 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7faf5443feb0] 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 13: __libc_start_main() 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 14: _start() 2024-02-10T01:39:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:39:06.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2024-02-10T01:39:06.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: -9999> 2024-02-10T01:28:58.548+0000 7faf51066640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59960) UID: 1000 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: -9998> 2024-02-10T01:39:06.692+0000 7faf54625640 -1 *** Caught signal (Aborted) ** 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: in thread 7faf54625640 thread_name:rbd-mirror 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7faf54454db0] 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7faf5449c39a] 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 3: pthread_cond_wait() 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faf548d5500] 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 5: rbd-mirror(+0x36bd2b) [0x55d31872ad2b] 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55d31873b2a3] 2024-02-10T01:39:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55d31872e5d0] 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 8: rbd-mirror(+0x9896f8) [0x55d318d486f8] 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 9: rbd-mirror(+0x9896e0) [0x55d318d486e0] 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55d31872caea] 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 11: main() 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7faf5443feb0] 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 13: __libc_start_main() 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 14: _start() 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-10T01:39:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr: 2024-02-10T01:39:07.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi060.stderr:daemon-helper: command crashed with signal 6 2024-02-10T01:39:08.389 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2024-02-10T01:39:08.390 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-02-10T01:39:08.893 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2024-02-10T01:39:08.893 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2024-02-10T01:39:08.893 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2024-02-10T01:39:08.894 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2024-02-10T01:39:08.894 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2024-02-10T01:39:08.895 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:39:08.894+0000 7fa49cb9f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 33313) UID: 0 2024-02-10T01:39:08.895 INFO:tasks.ceph.cluster1.mgr.x.smithi008.stderr:2024-02-10T01:39:08.894+0000 7fa49cb9f640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T01:39:08.895 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:39:08.894+0000 7feaac3e2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 33648) UID: 0 2024-02-10T01:39:08.895 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:39:08.894+0000 7fc9b5e13640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 33643) UID: 0 2024-02-10T01:39:08.896 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:39:08.894+0000 7fc9b5e13640 -1 osd.0 266 *** Got signal Terminated *** 2024-02-10T01:39:08.896 INFO:tasks.ceph.cluster1.osd.0.smithi008.stderr:2024-02-10T01:39:08.894+0000 7fc9b5e13640 -1 osd.0 266 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T01:39:08.896 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:39:08.894+0000 7feaac3e2640 -1 osd.2 266 *** Got signal Terminated *** 2024-02-10T01:39:08.896 INFO:tasks.ceph.cluster1.osd.2.smithi008.stderr:2024-02-10T01:39:08.894+0000 7feaac3e2640 -1 osd.2 266 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T01:39:08.896 INFO:tasks.ceph.cluster1.mon.a.smithi008.stderr:2024-02-10T01:39:08.894+0000 7f63130ac640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 33311) UID: 0 2024-02-10T01:39:08.896 INFO:tasks.ceph.cluster1.mon.a.smithi008.stderr:2024-02-10T01:39:08.894+0000 7f63130ac640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T01:39:08.897 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:39:08.894+0000 7f78be611640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 33644) UID: 0 2024-02-10T01:39:08.897 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:39:08.894+0000 7f78be611640 -1 osd.1 266 *** Got signal Terminated *** 2024-02-10T01:39:08.897 INFO:tasks.ceph.cluster1.osd.1.smithi008.stderr:2024-02-10T01:39:08.894+0000 7f78be611640 -1 osd.1 266 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T01:42:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T01:42:38.462+0000 7f406cc8f640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T01:42:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T01:42:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:42:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 service status 2024-02-10T01:47:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T01:47:38.550+0000 7fd3d94f6640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T01:47:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T01:47:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:47:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:47:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-02-10T01:47:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T01:47:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror images 2024-02-10T01:47:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror images' 2024-02-10T01:47:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2024-02-10T01:52:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T01:52:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (110) Connection timed out 2024-02-10T01:52:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:52:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror mirror pool info 2024-02-10T01:52:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:52:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror mirror pool info' 2024-02-10T01:52:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2024-02-10T01:57:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T01:57:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T01:57:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror mirror pool status 2024-02-10T01:57:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T01:57:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror mirror pool status' 2024-02-10T01:57:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T01:57:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2024-02-10T02:02:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T02:02:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:02:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:02:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls 2024-02-10T02:07:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror rbd_mirroring omap vals 2024-02-10T02:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror rbd_mirroring omap vals' 2024-02-10T02:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2024-02-10T02:12:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T02:12:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror rbd_mirror_leader omap vals 2024-02-10T02:12:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2024-02-10T02:12:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-02-10T02:17:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T02:17:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:17:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:17:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T02:17:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror ns1 images 2024-02-10T02:17:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror ns1 images' 2024-02-10T02:17:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2024-02-10T02:22:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T02:22:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (110) Connection timed out 2024-02-10T02:22:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:22:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:22:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirrorns1 mirror pool info 2024-02-10T02:22:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirrorns1 mirror pool info' 2024-02-10T02:22:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2024-02-10T02:27:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T02:27:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:27:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:27:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirrorns1 mirror pool status 2024-02-10T02:27:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirrorns1 mirror pool status' 2024-02-10T02:27:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T02:27:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2024-02-10T02:32:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T02:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2024-02-10T02:37:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2024-02-10T02:37:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2024-02-10T02:37:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-02-10T02:42:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T02:42:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2024-02-10T02:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2024-02-10T02:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-02-10T02:47:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T02:47:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:47:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T02:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror ns2 images' 2024-02-10T02:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror ns2 images 2024-02-10T02:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2024-02-10T02:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T02:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (110) Connection timed out 2024-02-10T02:52:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:52:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirrorns2 mirror pool info 2024-02-10T02:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirrorns2 mirror pool info' 2024-02-10T02:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2024-02-10T02:57:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T02:57:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T02:57:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T02:57:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirrorns2 mirror pool status 2024-02-10T02:57:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirrorns2 mirror pool status' 2024-02-10T02:57:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T02:57:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2024-02-10T03:02:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T03:02:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:02:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:02:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2024-02-10T03:07:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2024-02-10T03:07:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2024-02-10T03:07:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-02-10T03:12:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T03:12:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2024-02-10T03:12:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2024-02-10T03:12:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-02-10T03:17:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T03:17:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:17:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:17:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-02-10T03:17:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T03:17:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent images 2024-02-10T03:17:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent images' 2024-02-10T03:17:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2024-02-10T03:22:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T03:22:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (110) Connection timed out 2024-02-10T03:22:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:22:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:22:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent mirror pool info 2024-02-10T03:22:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent mirror pool info' 2024-02-10T03:22:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2024-02-10T03:27:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T03:27:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:27:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:27:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent mirror pool status 2024-02-10T03:27:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent mirror pool status' 2024-02-10T03:27:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T03:27:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2024-02-10T03:32:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T03:32:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:32:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:32:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2024-02-10T03:37:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2024-02-10T03:37:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2024-02-10T03:37:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-02-10T03:42:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T03:42:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2024-02-10T03:42:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2024-02-10T03:42:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-02-10T03:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T03:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T03:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent ns1 images 2024-02-10T03:47:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent ns1 images' 2024-02-10T03:47:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2024-02-10T03:52:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T03:52:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (110) Connection timed out 2024-02-10T03:52:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:52:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:52:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parentns1 mirror pool info 2024-02-10T03:52:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool info' 2024-02-10T03:52:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2024-02-10T03:57:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T03:57:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T03:57:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T03:57:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parentns1 mirror pool status 2024-02-10T03:57:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool status' 2024-02-10T03:57:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T03:57:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-02-10T04:02:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T04:02:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T04:02:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T04:02:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2024-02-10T04:07:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2024-02-10T04:07:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirroring omap vals' 2024-02-10T04:07:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-02-10T04:12:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T04:12:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2024-02-10T04:12:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-02-10T04:12:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-02-10T04:17:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T04:17:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T04:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T04:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-02-10T04:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent ns2 images 2024-02-10T04:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent ns2 images' 2024-02-10T04:17:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2024-02-10T04:22:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T04:22:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: listing images failed: (110) Connection timed out 2024-02-10T04:22:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T04:22:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T04:22:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool info' 2024-02-10T04:22:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parentns2 mirror pool info 2024-02-10T04:22:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2024-02-10T04:27:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T04:27:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T04:27:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T04:27:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parentns2 mirror pool status 2024-02-10T04:27:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool status' 2024-02-10T04:27:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T04:27:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-02-10T04:32:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:rbd: couldn't connect to the cluster! 2024-02-10T04:32:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T04:32:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T04:32:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2024-02-10T04:37:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2024-02-10T04:37:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirroring omap vals' 2024-02-10T04:37:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-02-10T04:42:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T04:42:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2024-02-10T04:42:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-02-10T04:42:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-02-10T04:47:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-02-10T04:47:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-02-10T04:47:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 2024-02-10T04:47:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret 2024-02-10T04:47:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-02-10T04:47:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 3 2024-02-10T04:47:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster1:0 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-02-10T04:47:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T04:47:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T04:47:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T04:47:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T04:47:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-02-10T04:47:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-02-10T04:47:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-02-10T04:47:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ']' 2024-02-10T04:47:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist)' 2024-02-10T04:47:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster1:1 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:1 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:1 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T04:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T04:47:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster1:1 ']' 2024-02-10T04:47:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T04:47:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T04:47:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T04:47:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T04:47:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-02-10T04:47:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-02-10T04:47:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-02-10T04:47:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ']' 2024-02-10T04:47:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist)' 2024-02-10T04:47:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2024-02-10T04:47:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster1:2 2024-02-10T04:47:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:2 2024-02-10T04:47:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:2 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster1:2 ']' 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T04:47:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T04:47:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2024-02-10T04:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2024-02-10T04:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2024-02-10T04:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ']' 2024-02-10T04:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist)' 2024-02-10T04:47:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster1:3 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:3 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:3 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster1:3 ']' 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T04:47:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2024-02-10T04:47:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2024-02-10T04:47:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2024-02-10T04:47:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ']' 2024-02-10T04:47:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist)' 2024-02-10T04:47:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2024-02-10T04:47:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 3 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster2:0 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2:0 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:0 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T04:47:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-02-10T04:47:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster2:0 ']' 2024-02-10T04:47:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster2 2024-02-10T04:47:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster2 2024-02-10T04:47:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-02-10T04:47:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-02-10T04:47:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-02-10T04:47:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-02-10T04:47:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-02-10T04:47:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ']' 2024-02-10T04:47:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist)' 2024-02-10T04:47:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2024-02-10T04:47:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster2:1 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2:1 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:1 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=1 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 1 = cluster2:1 ']' 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster2 2024-02-10T04:47:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster2 2024-02-10T04:47:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=1 2024-02-10T04:47:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=1 2024-02-10T04:47:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-02-10T04:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-02-10T04:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-02-10T04:47:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ']' 2024-02-10T04:47:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist)' 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster2:2 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2:2 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:2 cluster instance 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 2 = cluster2:2 ']' 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=2 2024-02-10T04:47:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2024-02-10T04:47:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2024-02-10T04:47:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2024-02-10T04:47:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ']' 2024-02-10T04:47:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2024-02-10T04:47:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist)' 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster2:3 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2:3 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:3 cluster instance 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:3 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=3 2024-02-10T04:47:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 3 = cluster2:3 ']' 2024-02-10T04:47:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster2 2024-02-10T04:47:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster2 2024-02-10T04:47:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=3 2024-02-10T04:47:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=3 2024-02-10T04:47:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2024-02-10T04:47:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2024-02-10T04:47:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2024-02-10T04:47:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ']' 2024-02-10T04:47:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist)' 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ continue 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z '' ']' 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster instance 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T04:47:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-02-10T04:52:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T04:52:39.947+0000 7f052d815640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T04:52:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T04:52:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T04:52:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-02-10T04:57:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T04:57:40.038+0000 7f2e6df63640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T04:57:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T04:57:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T04:57:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-02-10T05:02:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T05:02:40.128+0000 7f6f1b2ed640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T05:02:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T05:02:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-02-10T05:02:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-02-10T05:07:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-02-10T05:07:40.219+0000 7fb5f1630640 0 monclient(hunting): authenticate timed out after 300 2024-02-10T05:07:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T05:07:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-02-10T05:07:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirrors cluster1 2024-02-10T05:07:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-02-10T05:07:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 3 2024-02-10T05:07:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster1:0 '' 2024-02-10T05:07:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:0 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster1:1 '' 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:1 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster1:2 '' 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:2 2024-02-10T05:07:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster1:3 '' 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:3 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirrors cluster2 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 3 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster2:0 '' 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:0 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster2:1 '' 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:1 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster2:2 '' 2024-02-10T05:07:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:2 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster2:3 '' 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:3 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n 1 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z 1 ']' 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test '' = /tmp/tmp.ywJfcqGoUP 2024-02-10T05:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -Rf /tmp/tmp.ywJfcqGoUP 2024-02-10T05:07:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:FAIL 2024-02-10T05:07:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-02-10T05:07:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 110 -eq 0 ']' 2024-02-10T05:07:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo FAIL 2024-02-10T05:07:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ exit 110 2024-02-10T05:07:40.232 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-02-10T05:07:40.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-02-10T05:07:40.526 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi060 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-02-10T05:07:40.730 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=76ee3d9f257446519d683777ce9bf050 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi060 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-02-10T05:07:40.733 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-02-10T05:07:40.744 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-02-10T05:07:40.744 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-02-10T05:07:40.749 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-02-10T05:07:40.789 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-02-10T05:07:40.790 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-02-10T05:07:40.794 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:07:40.824 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-02-10T05:07:40.824 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:07:40.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi060.stderr:2024-02-10T05:07:40.824+0000 7fb77f7d5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 38876) UID: 1000 2024-02-10T05:12:39.889 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T05:12:39.897 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:12:39.912 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-02-10T05:12:39.912 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:12:39.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi060.stderr:2024-02-10T05:12:39.913+0000 7f479e7a4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 44547) UID: 1000 2024-02-10T05:17:38.981 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T05:17:38.988 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:17:39.000 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-02-10T05:17:39.000 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:17:39.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi060.stderr:2024-02-10T05:17:39.000+0000 7f94cf902640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 53375) UID: 1000 2024-02-10T05:22:38.075 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T05:22:38.083 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:22:38.138 ERROR:tasks.rbd_mirror.cluster2.client.mirror.0:tried to stop a non-running daemon 2024-02-10T05:22:38.141 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:22:38.152 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-02-10T05:22:38.153 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:22:38.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi060.stderr:2024-02-10T05:22:38.153+0000 7fbb907e4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 48223) UID: 1000 2024-02-10T05:27:37.218 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T05:27:37.228 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:27:37.244 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-02-10T05:27:37.245 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:27:37.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi060.stderr:2024-02-10T05:27:37.245+0000 7ff498c2d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 42591) UID: 1000 2024-02-10T05:32:36.325 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T05:32:36.332 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:32:36.347 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-02-10T05:32:36.347 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:32:36.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi060.stderr:2024-02-10T05:32:36.347+0000 7fa9ff070640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49666) UID: 1000 2024-02-10T05:37:35.425 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T05:37:35.432 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-02-10T05:37:35.444 ERROR:tasks.rbd_mirror.cluster1.client.mirror.0:tried to stop a non-running daemon 2024-02-10T05:37:35.448 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-10T05:37:35.487 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-02-10T05:37:35.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-02-10T05:39:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-10T05:39:35.530 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-02-10T05:39:35.545 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-10T05:39:35.546 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-10T05:39:35.546 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-02-10T05:39:35.546 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:39:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-10T05:39:35.549 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2024-02-10T05:39:35.563 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-02-10T05:39:35.563 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-02-10T05:39:35.563 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:39:35.563 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-02-10T05:39:35.563 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-02-10T05:39:35.563 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:39:35.564 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-02-10T05:39:35.564 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-10T05:39:35.564 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-02-10T05:39:35.564 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:39:35.564 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-02-10T05:39:35.564 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-10T05:39:35.565 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-02-10T05:39:35.565 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:39:35.565 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-02-10T05:39:35.565 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-10T05:39:35.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-02-10T05:39:35.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T05:39:35.609 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-02-10T05:39:35.868 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T05:39:35.868 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-02-10T05:39:36.064 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T05:39:36.064 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-02-10T05:39:36.236 INFO:tasks.ceph:Archiving mon data... 2024-02-10T05:39:36.238 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/data/cluster2.mon.a.tgz 2024-02-10T05:39:36.238 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-02-10T05:39:36.252 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.mfwiHlByt7 2024-02-10T05:39:36.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.mfwiHlByt7 2024-02-10T05:39:36.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.mfwiHlByt7 2024-02-10T05:39:36.497 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.mfwiHlByt7 is 381KB 2024-02-10T05:39:36.564 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.mfwiHlByt7 2024-02-10T05:39:36.579 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-10T05:39:36.580 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-02-10T05:39:36.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-02-10T05:39:36.645 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-02-10T05:39:36.659 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-10T05:39:36.676 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-02-10T05:39:36.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-02-10T05:41:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-10T05:41:36.718 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi008 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2024-02-10T05:41:36.734 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-10T05:41:36.734 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-10T05:41:36.734 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-02-10T05:41:36.734 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:41:36.738 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-02-10T05:41:36.739 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-02-10T05:41:36.739 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:41:36.739 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-02-10T05:41:36.739 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-02-10T05:41:36.740 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:41:36.740 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-02-10T05:41:36.740 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-10T05:41:36.740 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-02-10T05:41:36.740 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:41:36.740 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-02-10T05:41:36.741 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-10T05:41:36.741 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-02-10T05:41:36.741 INFO:teuthology.orchestra.run:waiting for 300 2024-02-10T05:41:36.741 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-02-10T05:41:36.741 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-10T05:41:36.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-02-10T05:41:36.780 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T05:41:36.780 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-02-10T05:41:36.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T05:41:36.885 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-02-10T05:41:36.941 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T05:41:36.941 DEBUG:teuthology.orchestra.run.smithi008:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-02-10T05:41:36.999 INFO:tasks.ceph:Archiving mon data... 2024-02-10T05:41:36.999 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/data/cluster1.mon.a.tgz 2024-02-10T05:41:36.999 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-02-10T05:41:37.014 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.dyX4NPodgF 2024-02-10T05:41:37.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.dyX4NPodgF 2024-02-10T05:41:37.156 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.dyX4NPodgF 2024-02-10T05:41:37.240 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.dyX4NPodgF is 308KB 2024-02-10T05:41:37.334 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.dyX4NPodgF 2024-02-10T05:41:37.347 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-10T05:41:37.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-02-10T05:41:37.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-02-10T05:41:37.467 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-10T05:41:37.485 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-10T05:41:37.486 INFO:tasks.ceph:Archiving crash dumps... 2024-02-10T05:41:37.488 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi008/crash 2024-02-10T05:41:37.488 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-10T05:41:37.534 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi060/crash 2024-02-10T05:41:37.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-10T05:41:37.616 INFO:tasks.ceph:Compressing logs... 2024-02-10T05:41:37.617 DEBUG:teuthology.orchestra.run.smithi008:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:41:37.619 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:41:37.658 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37224.log 2024-02-10T05:41:37.658 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37224.log.gz 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33553.log 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37308.log 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.33553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37388.log 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33553.log.gz 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37308.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-02-10T05:41:37.659 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37308.log.gz 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37388.log.gz 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37728.log 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37140.log 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37728.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37728.log.gz 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37266.log 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37140.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37140.log.gz 2024-02-10T05:41:37.660 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33718.log 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37266.log.gz 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37514.log 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.33718.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33718.log.gz 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39081.log 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37514.log.gz 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37516.log 2024-02-10T05:41:37.661 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.39081.log: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39081.log.gz 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/ceph.tmp-client.admin.28821.log 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37516.log: gzip -5 --verbose 0.0% -- /var/log/ceph/cluster2-mgr.x.log -- replaced with /var/log/ceph/cluster1-client.admin.37516.log.gz 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/ceph.tmp-client.admin.28821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37621.log 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28821.log.gz 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33473.log 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37621.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37621.log.gz 2024-02-10T05:41:37.662 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.33652.log 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.33473.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33473.log.gz 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37642.log 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.33652.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33652.log.gz 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.37182.log 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37642.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37642.log.gz 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37389.log 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37182.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37182.log.gz 2024-02-10T05:41:37.663 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37389.log.gz 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37390.log 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37622.log 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr: 88.9% -- replaced with /var/log/ceph/cluster1.log.gz 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37390.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37390.log.gz 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37770.log 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37622.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37622.log.gz 2024-02-10T05:41:37.664 INFO:teuthology.orchestra.run.smithi008.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.37812.log 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37770.log.gz 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37000.log 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33315.log 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37812.log.gz 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37000.log.gz 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.33315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33315.log.gz 2024-02-10T05:41:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33513.log 2024-02-10T05:41:37.666 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37084.log 2024-02-10T05:41:37.666 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.33513.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37515.log 2024-02-10T05:41:37.666 INFO:teuthology.orchestra.run.smithi008.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.33513.log.gz 2024-02-10T05:41:37.666 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37084.log.gz 2024-02-10T05:41:37.666 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cluster1-client.admin.37515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37515.log.gz 2024-02-10T05:41:37.672 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44081.log 2024-02-10T05:41:37.673 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56422.log 2024-02-10T05:41:37.673 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.44081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60761.log 2024-02-10T05:41:37.674 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50195.log 2024-02-10T05:41:37.674 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44081.log.gz 2024-02-10T05:41:37.674 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56422.log: /var/log/ceph/cluster1-client.mirror.60761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60263.log 2024-02-10T05:41:37.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50195.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.56422.log.gz /var/log/ceph/cluster1-client.mirror.54195.log 2024-02-10T05:41:37.677 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:37.677 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60761.log.gz 2024-02-10T05:41:37.677 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50195.log.gz 2024-02-10T05:41:37.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41441.log 2024-02-10T05:41:37.677 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60263.log.gz 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47462.log 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42379.log 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr: 81.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41441.log.gz 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45969.log 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59137.log 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47462.log.gz 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45969.log: gzip 92.2% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.54195.log.gz --verbose 2024-02-10T05:41:37.678 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster2-client.mirror.46860.log 2024-02-10T05:41:37.679 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50731.log 2024-02-10T05:41:37.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59137.log: /var/log/ceph/cluster2-client.mirror.46860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43328.log 2024-02-10T05:41:37.679 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59137.log.gz 2024-02-10T05:41:37.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47716.log 2024-02-10T05:41:37.679 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46860.log.gz 2024-02-10T05:41:37.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63567.log 2024-02-10T05:41:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45712.log 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63567.log: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51950.log 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47716.log.gz 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63567.log.gz 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62747.log 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51950.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45712.log.gz 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.mirror.45883.log 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62747.log: gzip -5 87.7% --verbose -- /var/log/ceph/cluster2-client.mirror.54637.log 2024-02-10T05:41:37.728 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62747.log.gz 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62827.log 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4%/var/log/ceph/cluster2-client.mirror.54637.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42379.log.gz 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45969.log.gz 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50016.log 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62827.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62827.log.gz 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.mirror.58695.log 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50016.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59772.log 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54637.log.gz 2024-02-10T05:41:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58695.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50016.log.gz 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60153.log 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43328.log.gz 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.39072.log 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58695.log.gz 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59772.log: gzip/var/log/ceph/cluster2-client.mirror.60153.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51812.log 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.39072.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60153.log.gz 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43371.log 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster2-client.mirror.51812.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57099.log 2024-02-10T05:41:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43371.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51812.log.gz 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64081.log 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59772.log.gz 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57099.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45883.log.gz 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41207.log 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64081.log: gzip -5 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64081.log.gz 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.50422.log 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41207.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57099.log.gz 2024-02-10T05:41:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57121.log 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50422.log: gzip -5 --verbose -- 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50422.log.gz 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/cluster1-client.mirror.45248.log 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58329.log 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45248.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57121.log.gz 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49527.log 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0%/var/log/ceph/cluster1-client.mirror.58329.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45248.log.gz 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62147.log 2024-02-10T05:41:37.732 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58329.log.gz 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 93.4% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.51950.log.gz --verbose 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.admin.38386.log 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49527.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43371.log.gz 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57400.log 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38386.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62147.log.gz 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49527.log.gz 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38386.log.gz 2024-02-10T05:41:37.733 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60809.log 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57400.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.39072.log.gz 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61187.log 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60809.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57400.log.gz 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47591.log 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60809.log.gz 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40632.log 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61187.log.gz 2024-02-10T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54679.log 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47591.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41207.log.gz 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62787.log 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54679.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40632.log.gz 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45392.log 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54679.log.gz 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41824.log 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53901.log 2024-02-10T05:41:37.735 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62787.log.gz 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41824.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45392.log.gz 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56263.log 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41824.log.gz 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48802.log 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53901.log.gz 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49039.log 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56263.log.gz 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55480.log 2024-02-10T05:41:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62647.log 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55480.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48802.log.gz 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63547.log 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49039.log.gz 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57689.log 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55480.log.gz 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62647.log.gz 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62627.log 2024-02-10T05:41:37.737 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63547.log.gz 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61287.log 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57689.log: /var/log/ceph/cluster1-client.mirror.62627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45797.log 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57689.log.gz 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61287.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62627.log.gz 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49850.log 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40653.log 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61287.log.gz 2024-02-10T05:41:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45797.log: /var/log/ceph/cluster2-client.mirror.49850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64843.log 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58808.log 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49850.log.gz/var/log/ceph/cluster2-client.mirror.64843.log: 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62927.log 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64843.log.gz 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56094.log 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42002.log 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.56094.log: -- replaced with /var/log/ceph/cluster1-client.mirror.62927.log.gz 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64285.log 2024-02-10T05:41:37.739 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56094.log.gz 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44564.log 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42002.log: /var/log/ceph/cluster1-client.admin.64285.log: gzip -5 50.4% -- replaced with /var/log/ceph/cluster1-client.admin.64285.log.gz 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58808.log.gz 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.55299.log 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43263.log 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37840.log 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42002.log.gz 2024-02-10T05:41:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37420.log 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55299.log.gz 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6%/var/log/ceph/cluster2-client.admin.37840.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43263.log.gz 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37840.log.gz 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45797.log.gz 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44879.log 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37420.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40653.log.gz 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47000.log 2024-02-10T05:41:37.741 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37420.log.gz 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44564.log.gz 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63447.log 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53011.log 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47000.log: /var/log/ceph/cluster1-client.mirror.63447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64431.log 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.53011.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47000.log.gz 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63447.log.gz 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58788.log 2024-02-10T05:41:37.742 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53011.log.gz 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64431.log.gz 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63147.log 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48144.log 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64870.log 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63147.log.gz 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48144.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48144.log.gz 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61107.log 2024-02-10T05:41:37.743 INFO:teuthology.orchestra.run.smithi060.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58788.log.gz 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49791.log 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64870.log.gz 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61227.log 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61107.log.gz 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61887.log 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55936.log 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61227.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49791.log.gz 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61887.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61227.log.gzgzip 2024-02-10T05:41:37.744 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62247.log 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61887.log.gz 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64985.log 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55936.log: /var/log/ceph/cluster1-client.mirror.62247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57340.log 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64985.log: 88.3%gzip -5 87.3% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.62247.log.gz /var/log/ceph/cluster1-client.mirror.42836.log 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.64985.log.gz 2024-02-10T05:41:37.745 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55936.log.gz 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50630.log 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44879.log.gz 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48653.log 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.50630.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42836.log.gz 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62267.log 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48653.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50630.log.gz 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56741.log 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58916.log 87.4% 2024-02-10T05:41:37.746 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48653.log.gz 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57340.log.gz 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62267.log.gz 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47101.log 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49367.log 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56741.log.gz 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47101.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58916.log.gz 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47208.log 2024-02-10T05:41:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49367.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47101.log.gz 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37754.log 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49367.log.gz 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44295.log 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47208.log: /var/log/ceph/cluster2-client.admin.37754.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37754.log.gz 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60609.log 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64704.log 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47208.log.gz 2024-02-10T05:41:37.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60609.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44295.log.gz 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61787.log 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64903.log 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64704.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64704.log.gz 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63727.log 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61787.log.gz 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43527.log 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64903.log.gz 2024-02-10T05:41:37.749 INFO:teuthology.orchestra.run.smithi060.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60609.log.gz 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41187.log 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52708.log 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63727.log.gz 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39327.log 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43527.log.gz 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57557.log 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42175.log 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39327.log.gz 2024-02-10T05:41:37.750 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52708.log.gz 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54130.log 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57557.log: /var/log/ceph/cluster2-client.mirror.42175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63947.log 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54130.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42175.log.gz 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51493.log 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54130.log.gz 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59482.log 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63947.log: /var/log/ceph/cluster2-client.mirror.51493.log: gzip 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63947.log.gz 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40945.log 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57557.log.gz 2024-02-10T05:41:37.751 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51493.log.gz 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58112.log 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59482.log.gz 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47040.log 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64685.log 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40945.log.gz 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58112.log.gz 2024-02-10T05:41:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63967.log 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41187.log.gz 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64685.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47040.log.gz 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50593.log 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64685.log.gz 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52276.log 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50593.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63967.log.gz 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41619.log 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50593.log.gz 2024-02-10T05:41:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38470.log 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52276.log: /var/log/ceph/cluster2-client.mirror.1.41619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64734.log 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52645.log 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38470.log.gz 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52276.log.gz 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.44564.log 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64734.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64734.log.gz 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45926.log 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52645.log.gz 2024-02-10T05:41:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.2.44564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61947.log 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56540.log 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45926.log: /var/log/ceph/cluster1-client.mirror.61947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61807.log 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.56540.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61947.log.gz 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56540.log.gz 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/cluster1-client.mirror.55323.log 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59591.log 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61807.log.gz 2024-02-10T05:41:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50969.log 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55323.log: /var/log/ceph/cluster2-client.mirror.59591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49648.log 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55323.log.gz 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50969.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59591.log.gz 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51572.log 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49648.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49648.log.gz 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47394.log 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39583.log 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47394.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51572.log.gz 2024-02-10T05:41:37.756 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47394.log.gz 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43191.log 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58940.log 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45926.log.gz 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr: 53.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39583.log.gz/var/log/ceph/cluster1-client.mirror.43191.log: 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42087.log 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58940.log: gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.43191.log.gz /var/log/ceph/cluster2-client.mirror.58350.log 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42087.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58940.log.gz 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42087.log.gz 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55371.log 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62107.log 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47983.log 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55371.log.gz 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62107.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62107.log.gz 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58350.log.gz 2024-02-10T05:41:37.778 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59977.log 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39072.log 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47983.log: /var/log/ceph/cluster2-client.mirror.59977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44247.log 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47983.log.gz 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39072.log: 82.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44856.log 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39072.log.gz 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44247.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44247.log.gz 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62027.log 2024-02-10T05:41:37.779 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59707.log 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44856.log: /var/log/ceph/cluster1-client.mirror.62027.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62027.log.gz 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59977.log.gz 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41441.log 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59009.log 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.41441.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59707.log.gz 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52296.log 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59009.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44856.log.gz 2024-02-10T05:41:37.780 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48121.log 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52296.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59009.log.gz 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52296.log.gz 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51436.log 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64098.log 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48121.log: /var/log/ceph/cluster2-client.mirror.51436.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41441.log.gz 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41619.log.gz 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48574.log 2024-02-10T05:41:37.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64341.log 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64098.log.gz 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47941.log 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48574.log.gz 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64341.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64341.log.gz 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39566.log 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41960.log 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39566.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47941.log.gz 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39566.log.gz 2024-02-10T05:41:37.782 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50868.log 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45820.log 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50868.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50868.log.gz 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41980.log 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45820.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41960.log.gz 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37879.log 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41980.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45820.log.gz 2024-02-10T05:41:37.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57665.log 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37879.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37879.log.gz 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50115.log 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57665.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57665.log.gz 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62847.log 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50115.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50115.log.gz 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40608.log 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62847.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62847.log.gz 2024-02-10T05:41:37.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48706.log 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40608.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40608.log.gz 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41980.log.gz 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62407.log 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46184.log 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48706.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48706.log.gz 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62047.log 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62407.log.gz 2024-02-10T05:41:37.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44719.log 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62047.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62047.log.gz 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57420.log 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44719.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44719.log.gz 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47963.log 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57420.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57420.log.gz 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53761.log 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47963.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46184.log.gz 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45106.log 2024-02-10T05:41:37.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53761.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53761.log.gz 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43471.log 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.45106.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.45106.log.gz 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43564.log 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.43471.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.43471.log.gz 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59731.log 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43564.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43564.log.gz 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50769.log 2024-02-10T05:41:37.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59731.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59731.log.gz 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64714.log 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50769.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50769.log.gz 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56492.log 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64714.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.admin.64714.log.gz 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61527.log 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56492.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56492.log.gz 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64452.log 2024-02-10T05:41:37.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61527.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61527.log.gz 2024-02-10T05:41:37.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58546.log 2024-02-10T05:41:37.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64452.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64452.log.gz 2024-02-10T05:41:37.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64379.log 2024-02-10T05:41:37.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58546.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58546.log.gz 2024-02-10T05:41:37.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47121.log 2024-02-10T05:41:37.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64379.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64379.log.gz 2024-02-10T05:41:37.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57579.log 2024-02-10T05:41:37.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47121.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47121.log.gz 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59029.log 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57579.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57579.log.gz 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49059.log 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59029.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59029.log.gz 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62187.log 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49059.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49059.log.gz 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46442.log 2024-02-10T05:41:37.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62187.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62187.log.gz 2024-02-10T05:41:37.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63387.log 2024-02-10T05:41:37.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46442.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46442.log.gz 2024-02-10T05:41:37.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50652.log 2024-02-10T05:41:37.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63387.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63387.log.gz 2024-02-10T05:41:37.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55412.log 2024-02-10T05:41:37.802 INFO:teuthology.orchestra.run.smithi008.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-02-10T05:41:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45754.log 2024-02-10T05:41:37.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55412.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55412.log.gz 2024-02-10T05:41:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64525.log 2024-02-10T05:41:37.823 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45754.log: 95.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.44564.log.gz 2024-02-10T05:41:37.827 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45754.log.gz 2024-02-10T05:41:37.827 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60496.log 2024-02-10T05:41:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64525.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64525.log.gz 2024-02-10T05:41:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57641.log 2024-02-10T05:41:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60496.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60496.log.gz 2024-02-10T05:41:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47228.log 2024-02-10T05:41:37.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57641.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57641.log.gz 2024-02-10T05:41:37.830 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63507.log 2024-02-10T05:41:37.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47228.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47228.log.gz 2024-02-10T05:41:37.831 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46336.log 2024-02-10T05:41:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63507.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63507.log.gz 2024-02-10T05:41:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41075.log 2024-02-10T05:41:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46336.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46336.log.gz 2024-02-10T05:41:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29091.log 2024-02-10T05:41:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41075.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41075.log.gz 2024-02-10T05:41:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52814.log 2024-02-10T05:41:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.29091.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29091.log.gz 2024-02-10T05:41:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59296.log 2024-02-10T05:41:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52814.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52814.log.gz 2024-02-10T05:41:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47532.log 2024-02-10T05:41:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59296.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59296.log.gz 2024-02-10T05:41:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50789.log 2024-02-10T05:41:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47532.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47532.log.gz 2024-02-10T05:41:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38916.log 2024-02-10T05:41:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50789.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50789.log.gz 2024-02-10T05:41:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50509.log 2024-02-10T05:41:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38916.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38916.log.gz 2024-02-10T05:41:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53512.log 2024-02-10T05:41:37.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50509.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50509.log.gz 2024-02-10T05:41:37.849 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.38772.log 2024-02-10T05:41:37.850 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53512.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53512.log.gz 2024-02-10T05:41:37.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64174.log 2024-02-10T05:41:37.851 INFO:teuthology.orchestra.run.smithi008.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-02-10T05:41:37.863 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.3.38772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46594.log 2024-02-10T05:41:37.864 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64174.log: 70.2% -- replaced with /var/log/ceph/cluster1-client.admin.64174.log.gz 2024-02-10T05:41:37.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46900.log 2024-02-10T05:41:37.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46594.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46594.log.gz 2024-02-10T05:41:37.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39532.log 2024-02-10T05:41:37.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46900.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46900.log.gz 2024-02-10T05:41:37.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39639.log 2024-02-10T05:41:37.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39532.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39532.log.gz 2024-02-10T05:41:37.895 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.38772.log.gz 2024-02-10T05:41:37.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41848.log 2024-02-10T05:41:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39639.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39639.log.gz 2024-02-10T05:41:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53035.log 2024-02-10T05:41:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41848.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41848.log.gz 2024-02-10T05:41:37.898 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58964.log 2024-02-10T05:41:37.899 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53035.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53035.log.gz 2024-02-10T05:41:37.900 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49081.log 2024-02-10T05:41:37.901 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58964.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58964.log.gz 2024-02-10T05:41:37.901 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45158.log 2024-02-10T05:41:37.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49081.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47591.log.gz 2024-02-10T05:41:37.915 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60669.log 2024-02-10T05:41:37.915 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.45158.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.45158.log.gz 2024-02-10T05:41:37.915 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52049.log 2024-02-10T05:41:37.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60669.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60669.log.gz 2024-02-10T05:41:37.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42199.log 2024-02-10T05:41:37.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52049.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52049.log.gz 2024-02-10T05:41:37.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55275.log 2024-02-10T05:41:37.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42199.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42199.log.gz 2024-02-10T05:41:37.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48079.log 2024-02-10T05:41:37.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55275.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55275.log.gz 2024-02-10T05:41:37.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39259.log 2024-02-10T05:41:37.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48079.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48079.log.gz 2024-02-10T05:41:37.922 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43899.log 2024-02-10T05:41:37.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39259.log: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.39259.log.gz 2024-02-10T05:41:37.924 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51515.log 2024-02-10T05:41:37.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43899.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43899.log.gz 2024-02-10T05:41:37.925 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47191.log 2024-02-10T05:41:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55777.log 2024-02-10T05:41:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47191.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47191.log.gz 2024-02-10T05:41:37.950 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50969.log.gz 2024-02-10T05:41:37.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46654.log 2024-02-10T05:41:37.952 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55777.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55777.log.gz 2024-02-10T05:41:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46035.log 2024-02-10T05:41:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46654.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46654.log.gz 2024-02-10T05:41:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54657.log 2024-02-10T05:41:37.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46035.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46035.log.gz 2024-02-10T05:41:37.960 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61767.log 2024-02-10T05:41:37.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54657.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54657.log.gz 2024-02-10T05:41:37.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47020.log 2024-02-10T05:41:37.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61767.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61767.log.gz 2024-02-10T05:41:37.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38006.log 2024-02-10T05:41:37.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47020.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47020.log.gz 2024-02-10T05:41:37.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47329.log 2024-02-10T05:41:37.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38006.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38006.log.gz 2024-02-10T05:41:37.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47921.log 2024-02-10T05:41:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47250.log 2024-02-10T05:41:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47921.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47921.log.gz 2024-02-10T05:41:37.994 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51105.log 2024-02-10T05:41:38.004 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45949.log 2024-02-10T05:41:38.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51105.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51105.log.gz 2024-02-10T05:41:38.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50237.log 2024-02-10T05:41:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38772.log 2024-02-10T05:41:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50237.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50237.log.gz 2024-02-10T05:41:38.031 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44340.log 2024-02-10T05:41:38.031 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.38772.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45949.log.gz 2024-02-10T05:41:38.038 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51436.log.gz 2024-02-10T05:41:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38176.log 2024-02-10T05:41:38.057 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44340.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44340.log.gz 2024-02-10T05:41:38.061 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49549.log 2024-02-10T05:41:38.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38176.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38176.log.gz 2024-02-10T05:41:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54279.log 2024-02-10T05:41:38.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49549.log: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38772.log.gz 2024-02-10T05:41:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62707.log 2024-02-10T05:41:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54279.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54279.log.gz 2024-02-10T05:41:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41756.log 2024-02-10T05:41:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62707.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62707.log.gz 2024-02-10T05:41:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39201.log 2024-02-10T05:41:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41756.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41756.log.gz 2024-02-10T05:41:38.121 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45416.log 2024-02-10T05:41:38.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39201.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39201.log.gz 2024-02-10T05:41:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52455.log 2024-02-10T05:41:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45416.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45416.log.gz 2024-02-10T05:41:38.145 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42316.log 2024-02-10T05:41:38.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52455.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52455.log.gz 2024-02-10T05:41:38.150 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47963.log.gz 2024-02-10T05:41:38.151 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52686.log 2024-02-10T05:41:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42316.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42316.log.gz 2024-02-10T05:41:38.157 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57951.log 2024-02-10T05:41:38.158 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52686.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50731.log.gz 2024-02-10T05:41:38.159 INFO:teuthology.orchestra.run.smithi060.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52686.log.gz 2024-02-10T05:41:38.160 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59936.log 2024-02-10T05:41:38.161 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57951.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57951.log.gz 2024-02-10T05:41:38.161 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64046.log 2024-02-10T05:41:38.163 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59936.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59936.log.gz 2024-02-10T05:41:38.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52344.log 2024-02-10T05:41:38.163 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64046.log: 48.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64046.log.gz 2024-02-10T05:41:38.165 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49665.log 2024-02-10T05:41:38.165 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52344.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52344.log.gz 2024-02-10T05:41:38.165 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49507.log 2024-02-10T05:41:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49665.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49665.log.gz 2024-02-10T05:41:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56765.log 2024-02-10T05:41:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49507.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49507.log.gz 2024-02-10T05:41:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64641.log 2024-02-10T05:41:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56765.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56765.log.gz 2024-02-10T05:41:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52573.log 2024-02-10T05:41:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64641.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.admin.64641.log.gz 2024-02-10T05:41:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43049.log 2024-02-10T05:41:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52573.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52573.log.gz 2024-02-10T05:41:38.173 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55139.log 2024-02-10T05:41:38.173 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.43049.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43049.log.gz 2024-02-10T05:41:38.173 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38811.log 2024-02-10T05:41:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55139.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55139.log.gz 2024-02-10T05:41:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60090.log 2024-02-10T05:41:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.38811.log: 74.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38811.log.gz 2024-02-10T05:41:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44471.log 2024-02-10T05:41:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60090.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60090.log.gz 2024-02-10T05:41:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60295.log 2024-02-10T05:41:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.44471.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44471.log.gz 2024-02-10T05:41:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51775.log 2024-02-10T05:41:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60295.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60295.log.gz 2024-02-10T05:41:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57537.log 2024-02-10T05:41:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51775.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51775.log.gz 2024-02-10T05:41:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63407.log 2024-02-10T05:41:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57537.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57537.log.gz 2024-02-10T05:41:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45324.log 2024-02-10T05:41:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63407.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63407.log.gz 2024-02-10T05:41:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58133.log 2024-02-10T05:41:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45324.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45324.log.gz 2024-02-10T05:41:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45505.log 2024-02-10T05:41:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58133.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58133.log.gz 2024-02-10T05:41:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49426.log 2024-02-10T05:41:38.197 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45505.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45505.log.gz 2024-02-10T05:41:38.197 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52130.log 2024-02-10T05:41:38.198 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49426.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49426.log.gz 2024-02-10T05:41:38.199 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55548.log 2024-02-10T05:41:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52130.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52130.log.gz 2024-02-10T05:41:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52925.log 2024-02-10T05:41:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55548.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55548.log.gz 2024-02-10T05:41:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57866.log 2024-02-10T05:41:38.202 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52925.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52925.log.gz 2024-02-10T05:41:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40983.log 2024-02-10T05:41:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57866.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57866.log.gz 2024-02-10T05:41:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51552.log 2024-02-10T05:41:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40983.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40983.log.gz 2024-02-10T05:41:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50910.log 2024-02-10T05:41:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51552.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51552.log.gz 2024-02-10T05:41:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58892.log 2024-02-10T05:41:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50910.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50910.log.gz 2024-02-10T05:41:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50890.log 2024-02-10T05:41:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.51164.log 2024-02-10T05:41:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54568.log 2024-02-10T05:41:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50890.log: /var/log/ceph/cluster1-client.mirror.0.51164.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58892.log.gz 2024-02-10T05:41:38.228 INFO:teuthology.orchestra.run.smithi060.stderr: 93.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45570.log 2024-02-10T05:41:38.232 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54568.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54568.log.gz 2024-02-10T05:41:38.232 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56022.log 2024-02-10T05:41:38.233 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45570.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45570.log.gz 2024-02-10T05:41:38.233 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.51164.log.gz 2024-02-10T05:41:38.237 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51515.log.gz 2024-02-10T05:41:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52433.log 2024-02-10T05:41:38.244 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63247.log/var/log/ceph/cluster1-client.mirror.56022.log: 2024-02-10T05:41:38.245 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56022.log.gz/var/log/ceph/cluster1-client.mirror.52433.log: 2024-02-10T05:41:38.246 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52433.log.gz 2024-02-10T05:41:38.246 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61427.log 2024-02-10T05:41:38.247 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63247.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63247.log.gz 2024-02-10T05:41:38.249 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43987.log 2024-02-10T05:41:38.249 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61427.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61427.log.gz 2024-02-10T05:41:38.249 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39659.log 2024-02-10T05:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43987.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43987.log.gz 2024-02-10T05:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59523.log 2024-02-10T05:41:38.257 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39659.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39659.log.gz 2024-02-10T05:41:38.257 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53532.log 2024-02-10T05:41:38.259 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59523.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59523.log.gz 2024-02-10T05:41:38.259 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47143.log 2024-02-10T05:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53532.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53532.log.gz 2024-02-10T05:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46614.log 2024-02-10T05:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47143.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47143.log.gz 2024-02-10T05:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59113.log 2024-02-10T05:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46614.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46614.log.gz 2024-02-10T05:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47611.log 2024-02-10T05:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59113.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59113.log.gz 2024-02-10T05:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58719.log 2024-02-10T05:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47611.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47611.log.gz 2024-02-10T05:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63667.log 2024-02-10T05:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58719.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58719.log.gz 2024-02-10T05:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40536.log 2024-02-10T05:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63667.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63667.log.gz 2024-02-10T05:41:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49606.log 2024-02-10T05:41:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40536.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40536.log.gz 2024-02-10T05:41:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64542.log 2024-02-10T05:41:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49606.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49606.log.gz 2024-02-10T05:41:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39622.log 2024-02-10T05:41:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64542.log.gz 2024-02-10T05:41:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48240.log 2024-02-10T05:41:38.274 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39622.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39622.log.gz 2024-02-10T05:41:38.274 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47374.log 2024-02-10T05:41:38.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48240.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50652.log.gz 2024-02-10T05:41:38.277 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42113.log 2024-02-10T05:41:38.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47374.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47374.log.gz 2024-02-10T05:41:38.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37628.log 2024-02-10T05:41:38.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42113.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42113.log.gz 2024-02-10T05:41:38.280 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59912.log 2024-02-10T05:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37628.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37628.log.gz 2024-02-10T05:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64490.log 2024-02-10T05:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59912.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59912.log.gz 2024-02-10T05:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55661.log 2024-02-10T05:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64490.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64490.log.gz 2024-02-10T05:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51631.log 2024-02-10T05:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55661.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55661.log.gz 2024-02-10T05:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57034.log 2024-02-10T05:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51631.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51631.log.gz 2024-02-10T05:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63227.log 2024-02-10T05:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57034.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57034.log.gz 2024-02-10T05:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56400.log 2024-02-10T05:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63227.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63227.log.gz 2024-02-10T05:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41027.log 2024-02-10T05:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56400.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56400.log.gz 2024-02-10T05:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59611.log 2024-02-10T05:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41027.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41027.log.gz 2024-02-10T05:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53146.log 2024-02-10T05:41:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59611.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59611.log.gz 2024-02-10T05:41:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44987.log 2024-02-10T05:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53146.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53146.log.gz 2024-02-10T05:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56900.log 2024-02-10T05:41:38.305 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44987.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44987.log.gz 2024-02-10T05:41:38.307 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50137.log 2024-02-10T05:41:38.307 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56900.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56900.log.gz 2024-02-10T05:41:38.308 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45528.log 2024-02-10T05:41:38.319 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50137.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48240.log.gz 2024-02-10T05:41:38.319 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37753.log 2024-02-10T05:41:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45528.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45528.log.gz 2024-02-10T05:41:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42988.log 2024-02-10T05:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37753.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37753.log.gz 2024-02-10T05:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61327.log 2024-02-10T05:41:38.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48960.log 2024-02-10T05:41:38.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61327.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61327.log.gz 2024-02-10T05:41:38.346 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47250.log.gz 2024-02-10T05:41:38.354 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41800.log 2024-02-10T05:41:38.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48960.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48960.log.gz 2024-02-10T05:41:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64997.log 2024-02-10T05:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41800.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41800.log.gz 2024-02-10T05:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48537.log 2024-02-10T05:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64997.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64997.log.gz 2024-02-10T05:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40320.log 2024-02-10T05:41:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48537.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48537.log.gz 2024-02-10T05:41:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52151.log 2024-02-10T05:41:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40320.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40320.log.gz 2024-02-10T05:41:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60864.log 2024-02-10T05:41:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52151.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52151.log.gz 2024-02-10T05:41:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49920.log 2024-02-10T05:41:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.60864.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60864.log.gz 2024-02-10T05:41:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59888.log 2024-02-10T05:41:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49920.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49920.log.gz 2024-02-10T05:41:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49490.log 2024-02-10T05:41:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59888.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59888.log.gz 2024-02-10T05:41:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46141.log 2024-02-10T05:41:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49490.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49490.log.gz 2024-02-10T05:41:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.46731.log 2024-02-10T05:41:38.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46141.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46141.log.gz 2024-02-10T05:41:38.373 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46121.log 2024-02-10T05:41:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.46731.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.46731.log.gz 2024-02-10T05:41:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59274.log 2024-02-10T05:41:38.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46121.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46121.log.gz 2024-02-10T05:41:38.387 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58671.log 2024-02-10T05:41:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59274.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59274.log.gz 2024-02-10T05:41:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53853.log 2024-02-10T05:41:38.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58671.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58671.log.gz 2024-02-10T05:41:38.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62347.log 2024-02-10T05:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53853.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53853.log.gz 2024-02-10T05:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56139.log 2024-02-10T05:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62347.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62347.log.gz 2024-02-10T05:41:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61007.log 2024-02-10T05:41:38.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56139.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56139.log.gz 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48822.log 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48121.log.gz 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61007.log: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51614.log 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61007.log.gz/var/log/ceph/cluster2-client.mirror.48822.log: 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48822.log.gz 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.47357.log 2024-02-10T05:41:38.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46399.log 2024-02-10T05:41:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51614.log: 79.8%/var/log/ceph/cluster2-client.mirror.47357.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51614.log.gz 2024-02-10T05:41:38.401 INFO:teuthology.orchestra.run.smithi060.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47357.log.gz 2024-02-10T05:41:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47758.log 2024-02-10T05:41:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64414.log 2024-02-10T05:41:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47758.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46399.log.gz 2024-02-10T05:41:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60239.log 2024-02-10T05:41:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64414.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64414.log.gz 2024-02-10T05:41:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58281.log 2024-02-10T05:41:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60239.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60239.log.gz 2024-02-10T05:41:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40889.log 2024-02-10T05:41:38.412 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58281.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58281.log.gz 2024-02-10T05:41:38.412 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64192.log 2024-02-10T05:41:38.412 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40889.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40889.log.gz 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54998.log 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64192.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64192.log.gz 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45368.log 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54998.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54998.log.gz 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48042.log 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45368.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45368.log.gz 2024-02-10T05:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37378.log 2024-02-10T05:41:38.425 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48042.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47329.log.gz 2024-02-10T05:41:38.425 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52238.log 2024-02-10T05:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43351.log/var/log/ceph/cluster2-client.admin.37378.log: 2024-02-10T05:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37378.log.gz 2024-02-10T05:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52238.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52238.log.gz 2024-02-10T05:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59503.log 2024-02-10T05:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43351.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43351.log.gz 2024-02-10T05:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56311.log 2024-02-10T05:41:38.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59503.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59503.log.gz 2024-02-10T05:41:38.437 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61207.log 2024-02-10T05:41:38.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56311.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56311.log.gz 2024-02-10T05:41:38.438 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57621.log 2024-02-10T05:41:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61207.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61207.log.gz 2024-02-10T05:41:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63687.log 2024-02-10T05:41:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57621.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57621.log.gz 2024-02-10T05:41:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64396.log 2024-02-10T05:41:38.442 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63687.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63687.log.gz 2024-02-10T05:41:38.443 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45344.log 2024-02-10T05:41:38.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64396.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.64396.log.gz 2024-02-10T05:41:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43308.log 2024-02-10T05:41:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45344.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45344.log.gz 2024-02-10T05:41:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52194.log 2024-02-10T05:41:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43308.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43308.log.gz 2024-02-10T05:41:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63747.log 2024-02-10T05:41:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52194.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52194.log.gz 2024-02-10T05:41:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44271.log 2024-02-10T05:41:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63747.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63747.log.gz 2024-02-10T05:41:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58040.log 2024-02-10T05:41:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44271.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44271.log.gz 2024-02-10T05:41:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61627.log 2024-02-10T05:41:38.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58040.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58040.log.gz 2024-02-10T05:41:38.456 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64029.log 2024-02-10T05:41:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61627.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61627.log.gz 2024-02-10T05:41:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47778.log 2024-02-10T05:41:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64029.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64029.log.gz 2024-02-10T05:41:38.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41872.log 2024-02-10T05:41:38.461 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47778.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47778.log.gz 2024-02-10T05:41:38.461 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.43670.log 2024-02-10T05:41:38.461 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41872.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41872.log.gz 2024-02-10T05:41:38.461 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58474.log 2024-02-10T05:41:38.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.43670.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49081.log.gz 2024-02-10T05:41:38.475 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64473.log 2024-02-10T05:41:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58474.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58474.log.gz 2024-02-10T05:41:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59864.log 2024-02-10T05:41:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64473.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64473.log.gz 2024-02-10T05:41:38.478 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51891.log 2024-02-10T05:41:38.479 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59864.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59864.log.gz 2024-02-10T05:41:38.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46465.log 2024-02-10T05:41:38.479 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51891.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51891.log.gz 2024-02-10T05:41:38.480 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52525.log 2024-02-10T05:41:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46465.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46465.log.gz 2024-02-10T05:41:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63467.log 2024-02-10T05:41:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52525.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52525.log.gz 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64230.log 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49549.log.gz 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40584.log 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64230.log.gz 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63467.log.gz 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57886.log 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46920.log 2024-02-10T05:41:38.492 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40584.log.gz 2024-02-10T05:41:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44360.log 2024-02-10T05:41:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46920.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57886.log.gz 2024-02-10T05:41:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48844.log 2024-02-10T05:41:38.493 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46920.log.gz 2024-02-10T05:41:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60111.log 2024-02-10T05:41:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48844.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44360.log.gz 2024-02-10T05:41:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58213.log 2024-02-10T05:41:38.498 INFO:teuthology.orchestra.run.smithi060.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.43670.log.gz 2024-02-10T05:41:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42401.log 2024-02-10T05:41:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58213.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58213.log.gz 2024-02-10T05:41:38.500 INFO:teuthology.orchestra.run.smithi060.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60111.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63047.log 2024-02-10T05:41:38.500 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51026.log 2024-02-10T05:41:38.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63047.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42401.log.gz 2024-02-10T05:41:38.501 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63047.log.gz 2024-02-10T05:41:38.501 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42608.log 2024-02-10T05:41:38.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65005.log 2024-02-10T05:41:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42608.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51026.log.gz 2024-02-10T05:41:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49937.log 2024-02-10T05:41:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.65005.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65005.log.gz 2024-02-10T05:41:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63187.log 2024-02-10T05:41:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49937.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49937.log.gz 2024-02-10T05:41:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42881.log 2024-02-10T05:41:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63187.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63187.log.gz 2024-02-10T05:41:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51913.log 2024-02-10T05:41:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42881.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42881.log.gz 2024-02-10T05:41:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57910.log 2024-02-10T05:41:38.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42924.log 2024-02-10T05:41:38.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57910.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57910.log.gz 2024-02-10T05:41:38.533 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50890.log.gz 2024-02-10T05:41:38.534 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47080.log 2024-02-10T05:41:38.534 INFO:teuthology.orchestra.run.smithi008.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-02-10T05:41:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42924.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42924.log.gz 2024-02-10T05:41:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49330.log 2024-02-10T05:41:38.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47080.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47080.log.gz 2024-02-10T05:41:38.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60949.log 2024-02-10T05:41:38.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49330.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49330.log.gz 2024-02-10T05:41:38.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64303.log 2024-02-10T05:41:38.549 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.60949.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42608.log.gz 2024-02-10T05:41:38.551 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-02-10T05:41:38.551 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64303.log: 11.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64303.log.gz 2024-02-10T05:41:38.551 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54910.log 2024-02-10T05:41:38.554 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-osd.1.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.60949.log.gz 2024-02-10T05:41:38.555 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53781.log 2024-02-10T05:41:38.556 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54910.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54910.log.gz 2024-02-10T05:41:38.558 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54438.log 2024-02-10T05:41:38.558 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53781.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53781.log.gz 2024-02-10T05:41:38.559 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38653.log 2024-02-10T05:41:38.559 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54438.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54438.log.gz 2024-02-10T05:41:38.560 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63887.log 2024-02-10T05:41:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.38653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63767.log 2024-02-10T05:41:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63887.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63887.log.gz 2024-02-10T05:41:38.584 INFO:teuthology.orchestra.run.smithi060.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38653.log.gz 2024-02-10T05:41:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61547.log 2024-02-10T05:41:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63767.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63767.log.gz 2024-02-10T05:41:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47482.log 2024-02-10T05:41:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61547.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61547.log.gz 2024-02-10T05:41:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51733.log 2024-02-10T05:41:38.589 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47482.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47482.log.gz 2024-02-10T05:41:38.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54978.log 2024-02-10T05:41:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51733.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51733.log.gz 2024-02-10T05:41:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56612.log 2024-02-10T05:41:38.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54978.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54978.log.gz 2024-02-10T05:41:38.592 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59434.log 2024-02-10T05:41:38.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56612.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56612.log.gz 2024-02-10T05:41:38.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42057.log 2024-02-10T05:41:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59434.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59434.log.gz 2024-02-10T05:41:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58743.log 2024-02-10T05:41:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42057.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42057.log.gz 2024-02-10T05:41:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46227.log 2024-02-10T05:41:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58743.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58743.log.gz 2024-02-10T05:41:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48181.log 2024-02-10T05:41:38.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46227.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46227.log.gz 2024-02-10T05:41:38.604 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62067.log 2024-02-10T05:41:38.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48181.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48181.log.gz 2024-02-10T05:41:38.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46960.log 2024-02-10T05:41:38.606 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62067.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62067.log.gz 2024-02-10T05:41:38.606 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52621.log 2024-02-10T05:41:38.608 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46960.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46960.log.gz 2024-02-10T05:41:38.608 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39481.log 2024-02-10T05:41:38.609 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52621.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52621.log.gz 2024-02-10T05:41:38.609 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52392.log 2024-02-10T05:41:38.610 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39481.log: 67.4% -- replaced with /var/log/ceph/cluster2-client.admin.39481.log.gz 2024-02-10T05:41:38.610 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46940.log 2024-02-10T05:41:38.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52392.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52392.log.gz 2024-02-10T05:41:38.612 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56359.log 2024-02-10T05:41:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46940.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46940.log.gz 2024-02-10T05:41:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64810.log 2024-02-10T05:41:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56359.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56359.log.gz 2024-02-10T05:41:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44836.log 2024-02-10T05:41:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64810.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64810.log.gz 2024-02-10T05:41:38.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62427.log 2024-02-10T05:41:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44836.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44836.log.gz 2024-02-10T05:41:38.622 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49126.log 2024-02-10T05:41:38.622 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62427.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62427.log.gz 2024-02-10T05:41:38.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49347.log 2024-02-10T05:41:38.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49126.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49126.log.gz 2024-02-10T05:41:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37462.log 2024-02-10T05:41:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49347.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49347.log.gz 2024-02-10T05:41:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55022.log 2024-02-10T05:41:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37462.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37462.log.gz 2024-02-10T05:41:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51280.log 2024-02-10T05:41:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55022.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55022.log.gz 2024-02-10T05:41:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52215.log 2024-02-10T05:41:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.51280.log: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51280.log.gz 2024-02-10T05:41:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59635.log 2024-02-10T05:41:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52215.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52215.log.gz 2024-02-10T05:41:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.49126.log 2024-02-10T05:41:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59635.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59635.log.gz 2024-02-10T05:41:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47166.log 2024-02-10T05:41:38.649 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.49126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64268.log 2024-02-10T05:41:38.649 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47166.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47166.log.gz 2024-02-10T05:41:38.668 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42247.log 2024-02-10T05:41:38.669 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64268.log: 57.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64268.log.gz 2024-02-10T05:41:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53392.log 2024-02-10T05:41:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42247.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42247.log.gz 2024-02-10T05:41:38.679 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47758.log.gz 2024-02-10T05:41:38.680 INFO:teuthology.orchestra.run.smithi060.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.49126.log.gz 2024-02-10T05:41:38.680 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40710.log 2024-02-10T05:41:38.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47060.log 2024-02-10T05:41:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.2.40710.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53392.log.gz 2024-02-10T05:41:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44527.log 2024-02-10T05:41:38.682 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47060.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47060.log.gz 2024-02-10T05:41:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40919.log 2024-02-10T05:41:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44527.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44527.log.gz 2024-02-10T05:41:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45906.log 2024-02-10T05:41:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40919.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40919.log.gz 2024-02-10T05:41:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51377.log 2024-02-10T05:41:38.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45906.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.40710.log.gz 2024-02-10T05:41:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63427.log 2024-02-10T05:41:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51377.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51377.log.gz 2024-02-10T05:41:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55094.log 2024-02-10T05:41:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63427.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63427.log.gz 2024-02-10T05:41:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51792.log 2024-02-10T05:41:38.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55094.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55094.log.gz 2024-02-10T05:41:38.690 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43105.log 2024-02-10T05:41:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51792.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51792.log.gz 2024-02-10T05:41:38.692 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45906.log.gz 2024-02-10T05:41:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47270.log 2024-02-10T05:41:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.43105.log: -- /var/log/ceph/cluster2-client.mirror.46293.log 2024-02-10T05:41:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47270.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43105.log.gz 2024-02-10T05:41:38.692 INFO:teuthology.orchestra.run.smithi060.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47270.log.gz 2024-02-10T05:41:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65177.log 2024-02-10T05:41:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55641.log 2024-02-10T05:41:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.65177.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65177.log.gz 2024-02-10T05:41:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-02-10T05:41:38.695 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55641.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55641.log.gz 2024-02-10T05:41:38.696 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62867.log 2024-02-10T05:41:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-mon.a.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46293.log.gz 2024-02-10T05:41:38.703 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.43564.log 2024-02-10T05:41:38.704 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62867.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62867.log.gz 2024-02-10T05:41:38.704 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49288.log 2024-02-10T05:41:38.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.2.43564.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.43564.log.gz 2024-02-10T05:41:38.707 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39498.log 2024-02-10T05:41:38.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49288.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49288.log.gz 2024-02-10T05:41:38.710 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59659.log 2024-02-10T05:41:38.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39498.log: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39498.log.gz 2024-02-10T05:41:38.710 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58450.log 2024-02-10T05:41:38.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59659.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59659.log.gz 2024-02-10T05:41:38.711 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56516.log 2024-02-10T05:41:38.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58450.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58450.log.gz 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58498.log 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56516.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56516.log.gz 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64008.log 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58498.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58498.log.gz 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64140.log 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64008.log.gz 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38344.log 2024-02-10T05:41:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64140.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64140.log.gz 2024-02-10T05:41:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60932.log 2024-02-10T05:41:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38344.log.gz 2024-02-10T05:41:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48161.log 2024-02-10T05:41:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60932.log: 56.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60932.log.gz 2024-02-10T05:41:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47416.log 2024-02-10T05:41:38.719 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48161.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48161.log.gz 2024-02-10T05:41:38.719 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53556.log 2024-02-10T05:41:38.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50848.log 2024-02-10T05:41:38.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53556.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53556.log.gz 2024-02-10T05:41:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41896.log 2024-02-10T05:41:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50848.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50848.log.gz 2024-02-10T05:41:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.53392.log 2024-02-10T05:41:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41896.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41896.log.gz 2024-02-10T05:41:38.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54217.log 2024-02-10T05:41:38.777 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.1.53392.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48844.log.gz 2024-02-10T05:41:38.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62947.log 2024-02-10T05:41:38.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54217.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54217.log.gz 2024-02-10T05:41:38.788 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48042.log.gz 2024-02-10T05:41:38.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42811.log 2024-02-10T05:41:38.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62947.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62947.log.gz 2024-02-10T05:41:38.790 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50568.log 2024-02-10T05:41:38.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42811.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42811.log.gz 2024-02-10T05:41:38.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46674.log 2024-02-10T05:41:38.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64507.log 2024-02-10T05:41:38.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46674.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46674.log.gz 2024-02-10T05:41:38.817 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64955.log 2024-02-10T05:41:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64507.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.64507.log.gz 2024-02-10T05:41:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39344.log 2024-02-10T05:41:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64955.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64955.log.gz 2024-02-10T05:41:38.836 INFO:teuthology.orchestra.run.smithi060.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.53392.log.gz 2024-02-10T05:41:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61367.log 2024-02-10T05:41:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39344.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39344.log.gz 2024-02-10T05:41:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61827.log 2024-02-10T05:41:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61367.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61367.log.gz 2024-02-10T05:41:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45640.log 2024-02-10T05:41:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61827.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61827.log.gz 2024-02-10T05:41:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61727.log 2024-02-10T05:41:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45640.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45640.log.gz 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55500.log 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61727.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61727.log.gz 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59161.log 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55500.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55500.log.gz 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44921.log 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59161.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59161.log.gz 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62087.log 2024-02-10T05:41:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44921.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44921.log.gz 2024-02-10T05:41:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59254.log 2024-02-10T05:41:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62087.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62087.log.gz 2024-02-10T05:41:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60589.log 2024-02-10T05:41:38.851 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59254.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59254.log.gz 2024-02-10T05:41:38.851 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-02-10T05:41:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60589.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60589.log.gz 2024-02-10T05:41:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50526.log 2024-02-10T05:41:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2.log: 88.4% -- replaced with /var/log/ceph/cluster2.log.gz 2024-02-10T05:41:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57320.log 2024-02-10T05:41:38.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50526.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50526.log.gz 2024-02-10T05:41:38.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40346.log 2024-02-10T05:41:38.861 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57320.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57320.log.gz 2024-02-10T05:41:38.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39225.log 2024-02-10T05:41:38.862 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40346.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40346.log.gz 2024-02-10T05:41:38.863 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46731.log 2024-02-10T05:41:38.864 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39225.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.39225.log.gz 2024-02-10T05:41:38.864 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54520.log 2024-02-10T05:41:38.865 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46731.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46731.log.gz 2024-02-10T05:41:38.866 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48723.log 2024-02-10T05:41:38.866 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54520.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54520.log.gz 2024-02-10T05:41:38.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45218.log 2024-02-10T05:41:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48723.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48723.log.gz 2024-02-10T05:41:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65061.log 2024-02-10T05:41:38.870 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.45218.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.45218.log.gz 2024-02-10T05:41:38.870 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52549.log 2024-02-10T05:41:38.871 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.65061.log: 9.2% -- replaced with /var/log/ceph/cluster1-client.admin.65061.log.gz 2024-02-10T05:41:38.871 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46207.log 2024-02-10T05:41:38.872 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52549.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52549.log.gz 2024-02-10T05:41:38.872 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43415.log 2024-02-10T05:41:38.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46207.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46207.log.gz 2024-02-10T05:41:38.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56046.log 2024-02-10T05:41:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51696.log 2024-02-10T05:41:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56046.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56046.log.gz 2024-02-10T05:41:38.896 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51913.log.gz 2024-02-10T05:41:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60340.log 2024-02-10T05:41:38.898 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50137.log.gz 2024-02-10T05:41:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51696.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51696.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63867.log 2024-02-10T05:41:38.898 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38671.log 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63867.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63867.log.gz 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43801.log 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.38671.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60340.log.gz 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39515.log 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43801.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43801.log.gz 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54259.log 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39515.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39515.log.gz 2024-02-10T05:41:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44319.log 2024-02-10T05:41:38.904 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54259.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54259.log.gz 2024-02-10T05:41:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46528.log 2024-02-10T05:41:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44319.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44319.log.gz 2024-02-10T05:41:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56335.log 2024-02-10T05:41:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46528.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46528.log.gz 2024-02-10T05:41:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41776.log 2024-02-10T05:41:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56335.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56335.log.gz 2024-02-10T05:41:38.914 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39395.log 2024-02-10T05:41:38.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41776.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41776.log.gz 2024-02-10T05:41:38.914 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41917.log 2024-02-10T05:41:38.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39395.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39395.log.gz 2024-02-10T05:41:38.915 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.50333.log 2024-02-10T05:41:38.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41917.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41917.log.gz 2024-02-10T05:41:38.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60888.log 2024-02-10T05:41:38.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.50333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48099.log 2024-02-10T05:41:38.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.60888.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60888.log.gz 2024-02-10T05:41:38.939 INFO:teuthology.orchestra.run.smithi060.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.50333.log.gz 2024-02-10T05:41:38.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33778.log 2024-02-10T05:41:38.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48099.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48099.log.gz 2024-02-10T05:41:38.943 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44199.log 2024-02-10T05:41:38.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.33778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33778.log.gz 2024-02-10T05:41:38.943 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38716.log 2024-02-10T05:41:38.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44199.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44199.log.gz 2024-02-10T05:41:38.944 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55046.log 2024-02-10T05:41:38.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.2.38716.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38716.log.gz 2024-02-10T05:41:38.947 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38218.log 2024-02-10T05:41:38.948 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55046.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55046.log.gz 2024-02-10T05:41:38.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61927.log 2024-02-10T05:41:38.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38218.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38218.log.gz 2024-02-10T05:41:38.950 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62307.log 2024-02-10T05:41:38.951 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61927.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61927.log.gz 2024-02-10T05:41:38.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42223.log 2024-02-10T05:41:38.952 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62307.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62307.log.gz 2024-02-10T05:41:38.952 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58522.log 2024-02-10T05:41:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42223.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42223.log.gz 2024-02-10T05:41:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51834.log 2024-02-10T05:41:38.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58522.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58522.log.gz 2024-02-10T05:41:38.955 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43849.log 2024-02-10T05:41:38.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60319.log 2024-02-10T05:41:38.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43849.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42988.log.gz 2024-02-10T05:41:38.969 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43849.log.gz 2024-02-10T05:41:38.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63307.log 2024-02-10T05:41:38.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60319.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60319.log.gz 2024-02-10T05:41:38.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52597.log 2024-02-10T05:41:38.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63307.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63307.log.gz 2024-02-10T05:41:38.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57227.log 2024-02-10T05:41:38.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52597.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52597.log.gz 2024-02-10T05:41:38.973 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50058.log 2024-02-10T05:41:38.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57227.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57227.log.gz 2024-02-10T05:41:38.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61087.log 2024-02-10T05:41:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54890.log 2024-02-10T05:41:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61087.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61087.log.gz 2024-02-10T05:41:38.998 INFO:teuthology.orchestra.run.smithi060.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38671.log.gz 2024-02-10T05:41:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48365.log 2024-02-10T05:41:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54890.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54890.log.gz 2024-02-10T05:41:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49022.log 2024-02-10T05:41:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48365.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48365.log.gz 2024-02-10T05:41:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45080.log 2024-02-10T05:41:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49022.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49022.log.gz 2024-02-10T05:41:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63587.log 2024-02-10T05:41:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.45080.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.45080.log.gz 2024-02-10T05:41:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64251.log 2024-02-10T05:41:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63587.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63587.log.gz 2024-02-10T05:41:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46379.log 2024-02-10T05:41:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64251.log: 11.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64251.log.gz 2024-02-10T05:41:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55434.log 2024-02-10T05:41:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46379.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46379.log.gz 2024-02-10T05:41:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60131.log 2024-02-10T05:41:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55434.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55434.log.gz 2024-02-10T05:41:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62687.log 2024-02-10T05:41:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60131.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60131.log.gz 2024-02-10T05:41:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33991.log 2024-02-10T05:41:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62687.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62687.log.gz 2024-02-10T05:41:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56588.log 2024-02-10T05:41:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.33991.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33991.log.gz 2024-02-10T05:41:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45664.log 2024-02-10T05:41:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56588.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56588.log.gz 2024-02-10T05:41:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44743.log 2024-02-10T05:41:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45664.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45664.log.gz 2024-02-10T05:41:39.026 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48385.log 2024-02-10T05:41:39.026 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44743.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44743.log.gz 2024-02-10T05:41:39.026 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60689.log 2024-02-10T05:41:39.027 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48385.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48385.log.gz 2024-02-10T05:41:39.028 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42501.log 2024-02-10T05:41:39.028 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60689.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60689.log.gz 2024-02-10T05:41:39.029 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38512.log 2024-02-10T05:41:39.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42501.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42501.log.gz 2024-02-10T05:41:39.030 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39276.log 2024-02-10T05:41:39.032 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38512.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38512.log.gz 2024-02-10T05:41:39.032 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50175.log 2024-02-10T05:41:39.032 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39276.log: 53.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39276.log.gz 2024-02-10T05:41:39.032 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50254.log 2024-02-10T05:41:39.033 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50175.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50175.log.gz 2024-02-10T05:41:39.033 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62967.log 2024-02-10T05:41:39.034 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50254.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50254.log.gz 2024-02-10T05:41:39.035 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58830.log 2024-02-10T05:41:39.036 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62967.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62967.log.gz 2024-02-10T05:41:39.037 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46313.log 2024-02-10T05:41:39.037 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58830.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58830.log.gz 2024-02-10T05:41:39.038 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39293.log 2024-02-10T05:41:39.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46313.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46313.log.gz 2024-02-10T05:41:39.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62887.log 2024-02-10T05:41:39.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39293.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39293.log.gz 2024-02-10T05:41:39.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47699.log 2024-02-10T05:41:39.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62887.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62887.log.gz 2024-02-10T05:41:39.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61167.log 2024-02-10T05:41:39.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47699.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47699.log.gz 2024-02-10T05:41:39.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62207.log 2024-02-10T05:41:39.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61167.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61167.log.gz 2024-02-10T05:41:39.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50095.log 2024-02-10T05:41:39.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62207.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62207.log.gz 2024-02-10T05:41:39.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64822.log 2024-02-10T05:41:39.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50095.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50095.log.gz 2024-02-10T05:41:39.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47287.log 2024-02-10T05:41:39.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64822.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64822.log.gz 2024-02-10T05:41:39.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52883.log 2024-02-10T05:41:39.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47287.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47287.log.gz 2024-02-10T05:41:39.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42787.log 2024-02-10T05:41:39.057 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52883.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52883.log.gz 2024-02-10T05:41:39.057 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56633.log 2024-02-10T05:41:39.058 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42787.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42787.log.gz 2024-02-10T05:41:39.058 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64778.log 2024-02-10T05:41:39.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56633.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56633.log.gz 2024-02-10T05:41:39.061 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.51280.log 2024-02-10T05:41:39.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64778.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.64778.log.gz 2024-02-10T05:41:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55070.log 2024-02-10T05:41:39.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.1.51280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49468.log 2024-02-10T05:41:39.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55070.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55070.log.gz 2024-02-10T05:41:39.077 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47416.log.gz 2024-02-10T05:41:39.078 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52505.log 2024-02-10T05:41:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49468.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.51280.log.gz 2024-02-10T05:41:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45440.log 2024-02-10T05:41:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52505.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52505.log.gz 2024-02-10T05:41:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62387.log 2024-02-10T05:41:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45440.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45440.log.gz 2024-02-10T05:41:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61607.log 2024-02-10T05:41:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62387.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62387.log.gz 2024-02-10T05:41:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34152.log 2024-02-10T05:41:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61607.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61607.log.gz 2024-02-10T05:41:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63647.log 2024-02-10T05:41:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.34152.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34152.log.gz 2024-02-10T05:41:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56878.log 2024-02-10T05:41:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63647.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63647.log.gz 2024-02-10T05:41:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44223.log 2024-02-10T05:41:39.088 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56878.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56878.log.gz 2024-02-10T05:41:39.088 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55873.log 2024-02-10T05:41:39.089 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44223.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44223.log.gz 2024-02-10T05:41:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44179.log 2024-02-10T05:41:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55873.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55873.log.gz 2024-02-10T05:41:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43781.log 2024-02-10T05:41:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44179.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44179.log.gz 2024-02-10T05:41:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60713.log 2024-02-10T05:41:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43781.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43781.log.gz 2024-02-10T05:41:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63207.log 2024-02-10T05:41:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60713.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60713.log.gz 2024-02-10T05:41:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64747.log 2024-02-10T05:41:39.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63207.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63207.log.gz 2024-02-10T05:41:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54082.log 2024-02-10T05:41:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64747.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64747.log.gz 2024-02-10T05:41:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57058.log 2024-02-10T05:41:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54082.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54082.log.gz 2024-02-10T05:41:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59386.log 2024-02-10T05:41:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57058.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57058.log.gz 2024-02-10T05:41:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59051.log 2024-02-10T05:41:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59386.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59386.log.gz 2024-02-10T05:41:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54351.log 2024-02-10T05:41:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59051.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59051.log.gz 2024-02-10T05:41:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52967.log 2024-02-10T05:41:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54351.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54351.log.gz 2024-02-10T05:41:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38653.log 2024-02-10T05:41:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52967.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52967.log.gz 2024-02-10T05:41:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61407.log 2024-02-10T05:41:39.116 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.38653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62167.log 2024-02-10T05:41:39.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61407.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61407.log.gz 2024-02-10T05:41:39.130 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58175.log 2024-02-10T05:41:39.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62167.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62167.log.gz 2024-02-10T05:41:39.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39549.log 2024-02-10T05:41:39.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58175.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58175.log.gz 2024-02-10T05:41:39.139 INFO:teuthology.orchestra.run.smithi060.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38653.log.gz 2024-02-10T05:41:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44425.log 2024-02-10T05:41:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39549.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39549.log.gz 2024-02-10T05:41:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64943.log 2024-02-10T05:41:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44425.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44425.log.gz 2024-02-10T05:41:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58587.log 2024-02-10T05:41:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64943.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64943.log.gz 2024-02-10T05:41:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58651.log 2024-02-10T05:41:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58587.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58587.log.gz 2024-02-10T05:41:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48240.log 2024-02-10T05:41:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58651.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58651.log.gz 2024-02-10T05:41:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53104.log 2024-02-10T05:41:39.158 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.3.48240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61667.log 2024-02-10T05:41:39.160 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53104.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53104.log.gz 2024-02-10T05:41:39.173 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56813.log 2024-02-10T05:41:39.174 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61667.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61667.log.gz 2024-02-10T05:41:39.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40182.log 2024-02-10T05:41:39.175 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56813.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56813.log.gz 2024-02-10T05:41:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43944.log 2024-02-10T05:41:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64563.log 2024-02-10T05:41:39.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43944.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64633.log 2024-02-10T05:41:39.218 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64563.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55572.log 2024-02-10T05:41:39.218 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.64563.log.gz 2024-02-10T05:41:39.218 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64633.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64633.log.gz 2024-02-10T05:41:39.222 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43944.log.gz 2024-02-10T05:41:39.226 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51164.log 2024-02-10T05:41:39.226 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55572.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55572.log.gz 2024-02-10T05:41:39.235 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49446.log 2024-02-10T05:41:39.244 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51164.log: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61267.log 2024-02-10T05:41:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49446.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54932.log 2024-02-10T05:41:39.254 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49446.log.gz 2024-02-10T05:41:39.255 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61267.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51164.log.gz 2024-02-10T05:41:39.260 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61647.log 2024-02-10T05:41:39.267 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54932.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61267.log.gz 2024-02-10T05:41:39.275 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54932.log.gz 2024-02-10T05:41:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62127.log 2024-02-10T05:41:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61647.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61647.log.gz 2024-02-10T05:41:39.282 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48923.log 2024-02-10T05:41:39.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62127.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62127.log.gz 2024-02-10T05:41:39.291 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48765.log 2024-02-10T05:41:39.310 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44029.log 2024-02-10T05:41:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48633.log 2024-02-10T05:41:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44029.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44029.log.gz 2024-02-10T05:41:39.331 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41404.log 2024-02-10T05:41:39.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48633.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48633.log.gz 2024-02-10T05:41:39.343 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49892.log 2024-02-10T05:41:39.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41404.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41404.log.gz 2024-02-10T05:41:39.353 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51834.log.gz 2024-02-10T05:41:39.354 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50217.log 2024-02-10T05:41:39.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49892.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50568.log.gz 2024-02-10T05:41:39.368 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55914.log 2024-02-10T05:41:39.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59342.log 2024-02-10T05:41:39.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55914.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55914.log.gz 2024-02-10T05:41:39.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44675.log 2024-02-10T05:41:39.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59342.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59342.log.gz 2024-02-10T05:41:39.403 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62367.log 2024-02-10T05:41:39.403 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44675.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44675.log.gz 2024-02-10T05:41:39.417 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38134.log 2024-02-10T05:41:39.417 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62367.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62367.log.gz 2024-02-10T05:41:39.430 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43923.log 2024-02-10T05:41:39.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38134.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38134.log.gz 2024-02-10T05:41:39.438 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50058.log.gz 2024-02-10T05:41:39.438 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38092.log 2024-02-10T05:41:39.440 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43923.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43923.log.gz 2024-02-10T05:41:39.440 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60568.log 2024-02-10T05:41:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38092.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38092.log.gz 2024-02-10T05:41:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46356.log 2024-02-10T05:41:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60568.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60568.log.gz 2024-02-10T05:41:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58088.log 2024-02-10T05:41:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46356.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46356.log.gz 2024-02-10T05:41:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64881.log 2024-02-10T05:41:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58088.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58088.log.gz 2024-02-10T05:41:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45485.log 2024-02-10T05:41:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64881.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64881.log.gz 2024-02-10T05:41:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62767.log 2024-02-10T05:41:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45485.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45485.log.gz 2024-02-10T05:41:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34090.log 2024-02-10T05:41:39.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62767.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62767.log.gz 2024-02-10T05:41:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53206.log 2024-02-10T05:41:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.34090.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34090.log.gz 2024-02-10T05:41:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57203.log 2024-02-10T05:41:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56181.log 2024-02-10T05:41:39.478 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57203.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57203.log.gz 2024-02-10T05:41:39.480 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40454.log 2024-02-10T05:41:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56181.log: 88.3% 93.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64757.log 2024-02-10T05:41:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40454.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53206.log.gz 2024-02-10T05:41:39.490 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40454.log.gz 2024-02-10T05:41:39.490 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56181.log.gz 2024-02-10T05:41:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62667.log 2024-02-10T05:41:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64757.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64757.log.gz 2024-02-10T05:41:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52903.log 2024-02-10T05:41:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62667.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62667.log.gz 2024-02-10T05:41:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53805.log 2024-02-10T05:41:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52903.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52903.log.gz 2024-02-10T05:41:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55596.log 2024-02-10T05:41:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53805.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53805.log.gz 2024-02-10T05:41:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46634.log 2024-02-10T05:41:39.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55596.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55596.log.gz 2024-02-10T05:41:39.501 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59844.log 2024-02-10T05:41:39.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46634.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46634.log.gz 2024-02-10T05:41:39.501 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56472.log 2024-02-10T05:41:39.502 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59844.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59844.log.gz 2024-02-10T05:41:39.502 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56380.log 2024-02-10T05:41:39.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56472.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56472.log.gz 2024-02-10T05:41:39.504 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56070.log 2024-02-10T05:41:39.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56380.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56380.log.gz 2024-02-10T05:41:39.506 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49002.log 2024-02-10T05:41:39.507 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56070.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56070.log.gz 2024-02-10T05:41:39.508 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40560.log 2024-02-10T05:41:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46551.log 2024-02-10T05:41:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40560.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40560.log.gz 2024-02-10T05:41:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54797.log 2024-02-10T05:41:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46551.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46551.log.gz 2024-02-10T05:41:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47628.log 2024-02-10T05:41:39.555 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54797.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54797.log.gz 2024-02-10T05:41:39.555 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57802.log 2024-02-10T05:41:39.556 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47628.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47628.log.gz 2024-02-10T05:41:39.573 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61487.log 2024-02-10T05:41:39.575 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57802.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57802.log.gz 2024-02-10T05:41:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61027.log 2024-02-10T05:41:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61487.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61487.log.gz 2024-02-10T05:41:39.605 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43415.log.gz 2024-02-10T05:41:39.607 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63927.log 2024-02-10T05:41:39.608 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61027.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61027.log.gz 2024-02-10T05:41:39.621 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37249.log 2024-02-10T05:41:39.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63927.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63927.log.gz 2024-02-10T05:41:39.634 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52086.log 2024-02-10T05:41:39.635 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37249.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37249.log.gz 2024-02-10T05:41:39.635 INFO:teuthology.orchestra.run.smithi060.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49468.log.gz 2024-02-10T05:41:39.654 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46485.log 2024-02-10T05:41:39.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52086.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52086.log.gz 2024-02-10T05:41:39.668 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56675.log 2024-02-10T05:41:39.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46485.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60520.log 2024-02-10T05:41:39.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56675.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56675.log.gz 2024-02-10T05:41:39.686 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46485.log.gz 2024-02-10T05:41:39.695 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43670.log 2024-02-10T05:41:39.695 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60520.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60520.log.gz 2024-02-10T05:41:39.700 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48923.log.gz 2024-02-10T05:41:39.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49586.log 2024-02-10T05:41:39.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55392.log 2024-02-10T05:41:39.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49586.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49586.log.gz 2024-02-10T05:41:39.733 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61907.log 2024-02-10T05:41:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55392.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55392.log.gz 2024-02-10T05:41:39.752 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43670.log.gz 2024-02-10T05:41:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64726.log 2024-02-10T05:41:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61907.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61907.log.gz 2024-02-10T05:41:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58430.log 2024-02-10T05:41:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64726.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64726.log.gz 2024-02-10T05:41:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48785.log 2024-02-10T05:41:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58430.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50217.log.gz 2024-02-10T05:41:39.791 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58430.log.gz 2024-02-10T05:41:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41144.log 2024-02-10T05:41:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48785.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48785.log.gz 2024-02-10T05:41:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44899.log 2024-02-10T05:41:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41144.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41144.log.gz 2024-02-10T05:41:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33951.log 2024-02-10T05:41:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44899.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44899.log.gz 2024-02-10T05:41:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48510.log 2024-02-10T05:41:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.33951.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33951.log.gz 2024-02-10T05:41:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62447.log 2024-02-10T05:41:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48510.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48765.log.gz 2024-02-10T05:41:39.822 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40428.log 2024-02-10T05:41:39.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62447.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62447.log.gz 2024-02-10T05:41:39.823 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61127.log 2024-02-10T05:41:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40428.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40428.log.gz 2024-02-10T05:41:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64623.log 2024-02-10T05:41:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61127.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61127.log.gz 2024-02-10T05:41:39.827 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39361.log 2024-02-10T05:41:39.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64623.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64623.log.gz 2024-02-10T05:41:39.829 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41348.log 2024-02-10T05:41:39.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39361.log: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39361.log.gz 2024-02-10T05:41:39.829 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56837.log 2024-02-10T05:41:39.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41348.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41348.log.gz 2024-02-10T05:41:39.831 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42860.log 2024-02-10T05:41:39.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56837.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56837.log.gz 2024-02-10T05:41:39.832 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63027.log 2024-02-10T05:41:39.834 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42860.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42860.log.gz 2024-02-10T05:41:39.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59977.log 2024-02-10T05:41:39.834 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63027.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63027.log.gz 2024-02-10T05:41:39.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54592.log 2024-02-10T05:41:39.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.59977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51713.log 2024-02-10T05:41:39.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54592.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54592.log.gz 2024-02-10T05:41:39.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63527.log 2024-02-10T05:41:39.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51713.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51713.log.gz 2024-02-10T05:41:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63267.log 2024-02-10T05:41:39.870 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63527.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63527.log.gz 2024-02-10T05:41:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56159.log 2024-02-10T05:41:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38716.log 2024-02-10T05:41:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56159.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63267.log.gz 2024-02-10T05:41:39.890 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56159.log.gz 2024-02-10T05:41:39.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-02-10T05:41:39.897 INFO:teuthology.orchestra.run.smithi060.stderr: 94.9% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49892.log.gz 2024-02-10T05:41:39.897 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.59977.log.gz 2024-02-10T05:41:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.38716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60737.log 82.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38716.log.gz 2024-02-10T05:41:39.906 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62807.log 2024-02-10T05:41:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55978.log 2024-02-10T05:41:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62807.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60737.log.gz 2024-02-10T05:41:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63827.log 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.55978.log: -- replaced with /var/log/ceph/cluster1-client.mirror.62807.log.gz 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48000.log 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.63827.log: -5 88.9% --verbose -- /var/log/ceph/cluster2-client.mirror.55231.log 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55978.log.gz 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63827.log.gz 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44137.log 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53922.log 2024-02-10T05:41:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44137.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48000.log.gz 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55231.log.gz 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61967.log 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44137.log.gz 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45274.log 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53922.log: /var/log/ceph/cluster1-client.mirror.61967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52666.log 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45274.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61967.log.gz 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64609.log 2024-02-10T05:41:39.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45274.log.gz 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52666.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53922.log.gz 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57492.log 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64609.log: 9.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42739.log 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.64609.log.gz 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52666.log.gz 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57492.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57492.log.gz 2024-02-10T05:41:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41099.log 2024-02-10T05:41:39.926 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.42739.log: --verbose -- /var/log/ceph/cluster1-client.mirror.62327.log 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61067.log 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42739.log.gz 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.62327.log: -- replaced with /var/log/ceph/cluster1-client.mirror.41099.log.gz 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50546.log 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62327.log.gz 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43147.log 2024-02-10T05:41:39.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50333.log 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50546.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61067.log.gz 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43147.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56219.log 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50333.log: -- replaced with /var/log/ceph/cluster2-client.mirror.50546.log.gz 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43147.log.gz 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50610.log 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45132.log 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56219.log: /var/log/ceph/cluster2-client.mirror.50610.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56219.log.gz 2024-02-10T05:41:39.928 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50610.log.gzgzip 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47569.log 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60472.log 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47569.log: /var/log/ceph/cluster1-client.admin.45132.log: 87.5% 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.45132.log.gz 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47569.log.gz 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62907.log 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64851.log 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60472.log.gz 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58153.log 2024-02-10T05:41:39.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62907.log.gz 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62987.log 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.64851.log.gz 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60448.log 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62987.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62987.log.gz 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54749.log 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58153.log.gz 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60448.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60448.log.gz 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62007.log 2024-02-10T05:41:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58064.log 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54749.log.gz 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7%/var/log/ceph/cluster1-client.mirror.62007.log: -- replaced with /var/log/ceph/cluster2-client.mirror.50333.log.gz 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62007.log.gz 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/cluster1-client.mirror.62567.log 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49999.log 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58064.log.gz 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52766.log 2024-02-10T05:41:39.931 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62567.log.gz 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49999.log: gzip 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49999.log.gz 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.65085.log 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46098.log 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3%/var/log/ceph/cluster1-client.admin.65085.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52766.log.gz 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr: 7.1% -- replaced with /var/log/ceph/cluster1-client.admin.65085.log.gz 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.42608.log 2024-02-10T05:41:39.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.46098.log: -- /var/log/ceph/cluster1-client.mirror.52368.log 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.2.42608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41619.log 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52368.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52368.log.gz 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53877.log 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41619.log: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41619.log.gz 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61687.log 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53877.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53877.log.gz 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54476.log 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61687.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61687.log.gz 2024-02-10T05:41:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54014.log 2024-02-10T05:41:39.934 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46098.log.gz 2024-02-10T05:41:39.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59458.log 2024-02-10T05:41:39.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54014.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54476.log.gz 2024-02-10T05:41:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46820.log 2024-02-10T05:41:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59458.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54014.log.gz 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60840.log 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59458.log.gz 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46820.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.60840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60840.log.gz 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46820.log.gz 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51006.log 2024-02-10T05:41:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55181.log 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51006.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51006.log.gz 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42557.log 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55181.log: 92.8% -- replaced with /var/log/ceph/cluster2.audit.log.gz 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55181.log.gz 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61747.log 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46012.log 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61747.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42557.log.gz 2024-02-10T05:41:39.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61747.log.gz 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49957.log 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53580.log 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46012.log: /var/log/ceph/cluster2-client.mirror.49957.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49957.log.gz 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41051.log 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53580.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53580.log.gz 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65074.log 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41051.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41051.log.gz 2024-02-10T05:41:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56239.log 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.65074.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.65074.log.gz 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56564.log 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56239.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56239.log.gz 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51394.log 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56564.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56564.log.gz 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46012.log.gz 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52173.log 2024-02-10T05:41:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51085.log 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52173.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52173.log.gz 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51394.log.gz 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56653.log 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59683.log 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51085.log: /var/log/ceph/cluster1-client.mirror.56653.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51085.log.gz 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60830.log 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59683.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59683.log.gz 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62547.log 2024-02-10T05:41:39.959 INFO:teuthology.orchestra.run.smithi060.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56653.log.gz 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38880.log 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62547.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60830.log.gz 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56962.log 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62547.log.gz 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49702.log 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56962.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38880.log.gz 2024-02-10T05:41:39.960 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62487.log 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56962.log.gz 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54303.log 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62487.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62487.log.gz 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57516.log 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54303.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54303.log.gz 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53604.log 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57516.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57516.log.gz 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64694.log 2024-02-10T05:41:39.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53604.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53604.log.gz 2024-02-10T05:41:39.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52838.log 2024-02-10T05:41:39.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64694.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64694.log.gz 2024-02-10T05:41:39.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64965.log 2024-02-10T05:41:39.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52838.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52838.log.gz 2024-02-10T05:41:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.51950.log 2024-02-10T05:41:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64965.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64965.log.gz 2024-02-10T05:41:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43501.log 2024-02-10T05:41:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.51950.log: 95.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48240.log.gz 2024-02-10T05:41:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53715.log 2024-02-10T05:41:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43501.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43501.log.gz 2024-02-10T05:41:39.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52413.log 2024-02-10T05:41:39.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53715.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53715.log.gz 2024-02-10T05:41:39.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54396.log 2024-02-10T05:41:39.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52413.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52413.log.gz 2024-02-10T05:41:39.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59209.log 2024-02-10T05:41:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54396.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54396.log.gz 2024-02-10T05:41:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51594.log 2024-02-10T05:41:39.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59209.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59209.log.gz 2024-02-10T05:41:39.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48203.log 2024-02-10T05:41:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51594.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.51950.log.gz 2024-02-10T05:41:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65111.log 2024-02-10T05:41:39.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48203.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49002.log.gz 2024-02-10T05:41:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54773.log 2024-02-10T05:41:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.65111.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.admin.65111.log.gz 2024-02-10T05:41:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43167.log 2024-02-10T05:41:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54773.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54773.log.gz 2024-02-10T05:41:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48596.log 2024-02-10T05:41:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43167.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43167.log.gz 2024-02-10T05:41:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64157.log 2024-02-10T05:41:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48596.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49702.log.gz 2024-02-10T05:41:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49870.log 2024-02-10T05:41:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64157.log: 49.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64157.log.gz 2024-02-10T05:41:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50481.log 2024-02-10T05:41:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49870.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49870.log.gz 2024-02-10T05:41:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51535.log 2024-02-10T05:41:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39378.log 2024-02-10T05:41:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51535.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51535.log.gz 2024-02-10T05:41:40.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54106.log 2024-02-10T05:41:40.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39378.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39378.log.gz 2024-02-10T05:41:40.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43079.log 2024-02-10T05:41:40.025 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54106.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54106.log.gz 2024-02-10T05:41:40.034 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50927.log 2024-02-10T05:41:40.035 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43079.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43079.log.gz 2024-02-10T05:41:40.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60785.log 2024-02-10T05:41:40.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50927.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50927.log.gz 2024-02-10T05:41:40.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57010.log 2024-02-10T05:41:40.062 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60785.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60785.log.gz 2024-02-10T05:41:40.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42024.log 2024-02-10T05:41:40.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57010.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57010.log.gz 2024-02-10T05:41:40.081 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38260.log 2024-02-10T05:41:40.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62507.log 2024-02-10T05:41:40.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38260.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38260.log.gz 2024-02-10T05:41:40.100 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40182.log.gz 2024-02-10T05:41:40.114 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58370.log 2024-02-10T05:41:40.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62507.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62507.log.gz 2024-02-10T05:41:40.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.44987.log 2024-02-10T05:41:40.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58370.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58370.log.gz 2024-02-10T05:41:40.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57782.log 2024-02-10T05:41:40.150 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.1.44987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50439.log 2024-02-10T05:41:40.151 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57782.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57782.log.gz 2024-02-10T05:41:40.162 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50036.log 2024-02-10T05:41:40.162 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50439.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50439.log.gz 2024-02-10T05:41:40.169 INFO:teuthology.orchestra.run.smithi060.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.44987.log.gz 2024-02-10T05:41:40.170 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42336.log 2024-02-10T05:41:40.171 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50036.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50036.log.gz 2024-02-10T05:41:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43964.log 2024-02-10T05:41:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42336.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64362.log 2024-02-10T05:41:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43964.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42336.log.gz 2024-02-10T05:41:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58233.log 2024-02-10T05:41:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64362.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64362.log.gz 2024-02-10T05:41:40.213 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43964.log.gz 2024-02-10T05:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37627.log 2024-02-10T05:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58233.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58233.log.gz 2024-02-10T05:41:40.232 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57251.log 2024-02-10T05:41:40.242 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37627.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45620.log 2024-02-10T05:41:40.242 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57251.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57251.log.gz 2024-02-10T05:41:40.245 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.37627.log.gz 2024-02-10T05:41:40.250 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51414.log 2024-02-10T05:41:40.260 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64596.log 2024-02-10T05:41:40.260 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51414.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51414.log.gz 2024-02-10T05:41:40.264 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45620.log.gz 2024-02-10T05:41:40.273 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63327.log 2024-02-10T05:41:40.274 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64596.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.64596.log.gz 2024-02-10T05:41:40.282 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46078.log 2024-02-10T05:41:40.296 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63327.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63327.log.gz 2024-02-10T05:41:40.296 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.mirror.44111.log 2024-02-10T05:41:40.305 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46078.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53652.log 2024-02-10T05:41:40.313 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44111.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46078.log.gz 2024-02-10T05:41:40.313 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48510.log.gz 2024-02-10T05:41:40.314 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41164.log 2024-02-10T05:41:40.314 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53652.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53652.log.gz 2024-02-10T05:41:40.314 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44111.log.gz 2024-02-10T05:41:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42271.log 2024-02-10T05:41:40.321 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41164.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41164.log.gz 2024-02-10T05:41:40.321 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45863.log 2024-02-10T05:41:40.321 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42271.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42271.log.gz 2024-02-10T05:41:40.322 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51068.log 2024-02-10T05:41:40.326 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45863.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51594.log.gz 2024-02-10T05:41:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51048.log 2024-02-10T05:41:40.329 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45863.log.gz 2024-02-10T05:41:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53059.log 2024-02-10T05:41:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51048.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51068.log.gz 2024-02-10T05:41:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33911.log 2024-02-10T05:41:40.330 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53059.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53059.log.gz 2024-02-10T05:41:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51673.log 2024-02-10T05:41:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.33911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33911.log.gz 2024-02-10T05:41:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45777.log 2024-02-10T05:41:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46571.log 2024-02-10T05:41:40.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45777.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50709.log 2024-02-10T05:41:40.357 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45777.log.gz 2024-02-10T05:41:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46571.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46571.log.gz 2024-02-10T05:41:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63287.log 2024-02-10T05:41:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50709.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50709.log.gz 2024-02-10T05:41:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57713.log 2024-02-10T05:41:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63287.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63287.log.gz 2024-02-10T05:41:40.398 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48864.log 2024-02-10T05:41:40.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57713.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65016.log 2024-02-10T05:41:40.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48864.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48864.log.gz 2024-02-10T05:41:40.411 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57713.log.gz 2024-02-10T05:41:40.415 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60631.log 2024-02-10T05:41:40.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.65016.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65016.log.gz 2024-02-10T05:41:40.424 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64663.log 2024-02-10T05:41:40.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60631.log: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42155.log 2024-02-10T05:41:40.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64663.log: -- replaced with /var/log/ceph/cluster2-client.mirror.60631.log.gz 2024-02-10T05:41:40.442 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44501.log 2024-02-10T05:41:40.446 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42155.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64663.log.gz 2024-02-10T05:41:40.448 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50481.log.gz 2024-02-10T05:41:40.449 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59362.log 2024-02-10T05:41:40.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44501.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44501.log.gz 2024-02-10T05:41:40.450 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47307.log 2024-02-10T05:41:40.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59362.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59362.log.gz 2024-02-10T05:41:40.452 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57468.log 2024-02-10T05:41:40.453 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47307.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47307.log.gz 2024-02-10T05:41:40.453 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49251.log 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57468.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57468.log.gz 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61987.log 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49251.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49251.log.gz 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52066.log 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61987.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42155.log.gz 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61987.log.gz 2024-02-10T05:41:40.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52746.log 2024-02-10T05:41:40.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52066.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52066.log.gz 2024-02-10T05:41:40.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46055.log 2024-02-10T05:41:40.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52746.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52746.log.gz 2024-02-10T05:41:40.460 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48675.log 2024-02-10T05:41:40.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46055.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46055.log.gz 2024-02-10T05:41:40.466 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44943.log 2024-02-10T05:41:40.468 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48675.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42024.log.gz 2024-02-10T05:41:40.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63167.log 2024-02-10T05:41:40.489 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54845.log 2024-02-10T05:41:40.490 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63167.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63167.log.gz 2024-02-10T05:41:40.495 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48596.log.gz 2024-02-10T05:41:40.496 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42763.log 2024-02-10T05:41:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54845.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54845.log.gz 2024-02-10T05:41:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54544.log 2024-02-10T05:41:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42763.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42763.log.gz 2024-02-10T05:41:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46250.log 2024-02-10T05:41:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54544.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54544.log.gz 2024-02-10T05:41:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41003.log 2024-02-10T05:41:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46250.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46250.log.gz 2024-02-10T05:41:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59093.log 2024-02-10T05:41:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41003.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41003.log.gz 2024-02-10T05:41:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42423.log 2024-02-10T05:41:40.509 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59093.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59093.log.gz 2024-02-10T05:41:40.509 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55524.log 2024-02-10T05:41:40.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43239.log 2024-02-10T05:41:40.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55524.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55524.log.gz 2024-02-10T05:41:40.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61147.log 2024-02-10T05:41:40.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43239.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43239.log.gz 2024-02-10T05:41:40.551 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56942.log 2024-02-10T05:41:40.551 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61147.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61147.log.gz 2024-02-10T05:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46164.log 2024-02-10T05:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56942.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56942.log.gz 2024-02-10T05:41:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48980.log 2024-02-10T05:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46164.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46164.log.gz 2024-02-10T05:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50752.log 2024-02-10T05:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48980.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48980.log.gz 2024-02-10T05:41:40.595 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48203.log.gz 2024-02-10T05:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48407.log 2024-02-10T05:41:40.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50752.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50752.log.gz 2024-02-10T05:41:40.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-02-10T05:41:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50672.log 2024-02-10T05:41:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-osd.2.log: gzip: /var/log/ceph/cluster1-client.mirror.2.42608.log: file size changed while zipping 2024-02-10T05:41:40.613 INFO:teuthology.orchestra.run.smithi060.stderr: 95.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.42608.log.gz 2024-02-10T05:41:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45840.log 2024-02-10T05:41:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50672.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50672.log.gz 2024-02-10T05:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58257.log 2024-02-10T05:41:40.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45840.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45840.log.gz 2024-02-10T05:41:40.654 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52108.log 2024-02-10T05:41:40.654 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58257.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58257.log.gz 2024-02-10T05:41:40.668 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51673.log.gz 2024-02-10T05:41:40.668 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63627.log 2024-02-10T05:41:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61467.log 2024-02-10T05:41:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63627.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63627.log.gz 2024-02-10T05:41:40.698 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63067.log 2024-02-10T05:41:40.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61467.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61467.log.gz 2024-02-10T05:41:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61387.log 2024-02-10T05:41:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63067.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63067.log.gz 2024-02-10T05:41:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50296.log 2024-02-10T05:41:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61387.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61387.log.gz 2024-02-10T05:41:40.725 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51048.log.gz 2024-02-10T05:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46694.log 2024-02-10T05:41:40.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58609.log 2024-02-10T05:41:40.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46694.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46694.log.gz 2024-02-10T05:41:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44403.log 2024-02-10T05:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58609.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58609.log.gz 2024-02-10T05:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55998.log 2024-02-10T05:41:40.769 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44403.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44403.log.gz 2024-02-10T05:41:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38596.log 2024-02-10T05:41:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55998.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55998.log.gz 2024-02-10T05:41:40.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43825.log 2024-02-10T05:41:40.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38596.log.gz 2024-02-10T05:41:40.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37752.log 2024-02-10T05:41:40.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43825.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43825.log.gz 2024-02-10T05:41:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38005.log 2024-02-10T05:41:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37752.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37752.log.gz 2024-02-10T05:41:40.815 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59185.log 2024-02-10T05:41:40.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38005.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38005.log.gz 2024-02-10T05:41:40.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38302.log 2024-02-10T05:41:40.834 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59185.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59185.log.gz 2024-02-10T05:41:40.839 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48675.log.gz 2024-02-10T05:41:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61847.log 2024-02-10T05:41:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38302.log.gz 2024-02-10T05:41:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54869.log 2024-02-10T05:41:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61847.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61847.log.gz 2024-02-10T05:41:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43874.log 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54869.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54869.log.gz 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60195.log 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43874.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43874.log.gz 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48554.log 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60195.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60195.log.gz 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37966.log 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48554.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48554.log.gz 2024-02-10T05:41:40.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64673.log 2024-02-10T05:41:40.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37966.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37966.log.gz 2024-02-10T05:41:40.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38839.log 2024-02-10T05:41:40.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64673.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64673.log.gz 2024-02-10T05:41:40.848 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38554.log 2024-02-10T05:41:40.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.1.38839.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38839.log.gz 2024-02-10T05:41:40.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37880.log 2024-02-10T05:41:40.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38554.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38554.log.gz 2024-02-10T05:41:40.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51854.log 2024-02-10T05:41:40.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37880.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37880.log.gz 2024-02-10T05:41:40.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46270.log 2024-02-10T05:41:40.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51854.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51854.log.gz 2024-02-10T05:41:40.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59752.log 2024-02-10T05:41:40.862 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46270.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46270.log.gz 2024-02-10T05:41:40.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49979.log 2024-02-10T05:41:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59752.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59752.log.gz 2024-02-10T05:41:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49813.log 2024-02-10T05:41:40.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57737.log 2024-02-10T05:41:40.888 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63487.log 2024-02-10T05:41:40.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48062.log 2024-02-10T05:41:40.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63487.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57737.log.gz 2024-02-10T05:41:40.896 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63487.log.gz 2024-02-10T05:41:40.922 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56287.log 2024-02-10T05:41:40.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48062.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48062.log.gz 2024-02-10T05:41:40.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.57951.log 2024-02-10T05:41:40.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56287.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56287.log.gz 2024-02-10T05:41:40.946 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42966.log 2024-02-10T05:41:40.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.57951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59410.log 2024-02-10T05:41:40.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42966.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42966.log.gz 2024-02-10T05:41:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53124.log 2024-02-10T05:41:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59410.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59410.log.gz 2024-02-10T05:41:40.976 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52108.log.gz 2024-02-10T05:41:40.996 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43215.log 2024-02-10T05:41:40.997 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53124.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53124.log.gz 2024-02-10T05:41:41.000 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44943.log.gz 2024-02-10T05:41:41.001 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39421.log 2024-02-10T05:41:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43215.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43215.log.gz 2024-02-10T05:41:41.003 INFO:teuthology.orchestra.run.smithi060.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.57951.log.gz 2024-02-10T05:41:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51456.log 2024-02-10T05:41:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39421.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39421.log.gz 2024-02-10T05:41:41.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40398.log 2024-02-10T05:41:41.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51456.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51456.log.gz 2024-02-10T05:41:41.006 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48468.log 2024-02-10T05:41:41.006 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40398.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40398.log.gz 2024-02-10T05:41:41.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53673.log 2024-02-10T05:41:41.008 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48468.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48468.log.gz 2024-02-10T05:41:41.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64769.log 2024-02-10T05:41:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53673.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53673.log.gz 2024-02-10T05:41:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42944.log 2024-02-10T05:41:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64769.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64769.log.gz 2024-02-10T05:41:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64320.log 2024-02-10T05:41:41.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42944.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42944.log.gz 2024-02-10T05:41:41.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37504.log 2024-02-10T05:41:41.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64320.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64320.log.gz 2024-02-10T05:41:41.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61567.log 2024-02-10T05:41:41.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37504.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37504.log.gz 2024-02-10T05:41:41.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40372.log 2024-02-10T05:41:41.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61567.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61567.log.gz 2024-02-10T05:41:41.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59794.log 2024-02-10T05:41:41.023 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40372.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40372.log.gz 2024-02-10T05:41:41.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51127.log 2024-02-10T05:41:41.025 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59794.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59794.log.gz 2024-02-10T05:41:41.025 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63907.log 2024-02-10T05:41:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50947.log 2024-02-10T05:41:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63907.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63907.log.gz 2024-02-10T05:41:41.055 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61047.log 2024-02-10T05:41:41.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44695.log 2024-02-10T05:41:41.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61047.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50947.log.gz 2024-02-10T05:41:41.074 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61047.log.gz 2024-02-10T05:41:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54154.log 2024-02-10T05:41:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44695.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44695.log.gz 2024-02-10T05:41:41.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55159.log 2024-02-10T05:41:41.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54154.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54154.log.gz 2024-02-10T05:41:41.112 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52320.log 2024-02-10T05:41:41.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55159.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55159.log.gz 2024-02-10T05:41:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44007.log 2024-02-10T05:41:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52320.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52320.log.gz 2024-02-10T05:41:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40673.log 2024-02-10T05:41:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44007.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48407.log.gz 2024-02-10T05:41:41.136 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44007.log.gz 2024-02-10T05:41:41.136 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37626.log 2024-02-10T05:41:41.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40673.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40673.log.gz 2024-02-10T05:41:41.145 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64063.log 2024-02-10T05:41:41.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37626.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37626.log.gz 2024-02-10T05:41:41.145 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61347.log 2024-02-10T05:41:41.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64063.log: 70.8% -- replaced with /var/log/ceph/cluster1-client.admin.64063.log.gz 2024-02-10T05:41:41.145 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41251.log 2024-02-10T05:41:41.146 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61347.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61347.log.gz 2024-02-10T05:41:41.146 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40492.log 2024-02-10T05:41:41.154 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49569.log 2024-02-10T05:41:41.154 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40492.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40492.log.gz 2024-02-10T05:41:41.166 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53083.log 2024-02-10T05:41:41.166 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49569.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49569.log.gz 2024-02-10T05:41:41.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48881.log 2024-02-10T05:41:41.174 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53083.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53083.log.gz 2024-02-10T05:41:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60544.log 2024-02-10T05:41:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48881.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48881.log.gz 2024-02-10T05:41:41.201 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56858.log 2024-02-10T05:41:41.201 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60544.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60544.log.gz 2024-02-10T05:41:41.210 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55733.log 2024-02-10T05:41:41.211 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56858.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56858.log.gz 2024-02-10T05:41:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64119.log 2024-02-10T05:41:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55733.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55733.log.gz 2024-02-10T05:41:41.236 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63707.log 2024-02-10T05:41:41.236 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64119.log.gz 2024-02-10T05:41:41.249 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60970.log 2024-02-10T05:41:41.249 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63707.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63707.log.gz 2024-02-10T05:41:41.270 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63847.log 2024-02-10T05:41:41.270 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60970.log: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60970.log.gz 2024-02-10T05:41:41.279 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64934.log 2024-02-10T05:41:41.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63847.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63847.log.gz 2024-02-10T05:41:41.297 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55347.log 2024-02-10T05:41:41.297 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64934.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64934.log.gz 2024-02-10T05:41:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42531.log 2024-02-10T05:41:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55347.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55347.log.gz 2024-02-10T05:41:41.317 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61307.log 2024-02-10T05:41:41.319 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42531.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42531.log.gz 2024-02-10T05:41:41.319 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63087.log 2024-02-10T05:41:41.320 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61307.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61307.log.gz 2024-02-10T05:41:41.330 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53628.log 2024-02-10T05:41:41.330 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63087.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63087.log.gz 2024-02-10T05:41:41.344 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42423.log.gz 2024-02-10T05:41:41.347 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48943.log 2024-02-10T05:41:41.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53628.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53628.log.gz 2024-02-10T05:41:41.347 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52790.log 2024-02-10T05:41:41.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48943.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48943.log.gz 2024-02-10T05:41:41.347 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64976.log 2024-02-10T05:41:41.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52790.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52790.log.gz 2024-02-10T05:41:41.351 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63807.log 2024-02-10T05:41:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64976.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64976.log.gz 2024-02-10T05:41:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51871.log 2024-02-10T05:41:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63807.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63807.log.gz 2024-02-10T05:41:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60382.log 2024-02-10T05:41:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51871.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51871.log.gz 2024-02-10T05:41:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40801.log 2024-02-10T05:41:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60382.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60382.log.gz 2024-02-10T05:41:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61507.log 2024-02-10T05:41:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40801.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40801.log.gz 2024-02-10T05:41:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53693.log 2024-02-10T05:41:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61507.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61507.log.gz 2024-02-10T05:41:41.356 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56986.log 2024-02-10T05:41:41.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53693.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53693.log.gz 2024-02-10T05:41:41.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50078.log 2024-02-10T05:41:41.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56986.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56986.log.gz 2024-02-10T05:41:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63007.log 2024-02-10T05:41:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50078.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50078.log.gz 2024-02-10T05:41:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49628.log 2024-02-10T05:41:41.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63007.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63007.log.gz 2024-02-10T05:41:41.362 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62287.log 2024-02-10T05:41:41.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46422.log 2024-02-10T05:41:41.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62287.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62287.log.gz 2024-02-10T05:41:41.377 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50296.log.gz 2024-02-10T05:41:41.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47815.log 2024-02-10T05:41:41.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46422.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46422.log.gz 2024-02-10T05:41:41.384 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46840.log 2024-02-10T05:41:41.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47815.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47815.log.gz 2024-02-10T05:41:41.386 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64862.log 2024-02-10T05:41:41.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46840.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46840.log.gz 2024-02-10T05:41:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54821.log 2024-02-10T05:41:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64862.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64862.log.gz 2024-02-10T05:41:41.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46980.log 2024-02-10T05:41:41.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54821.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54821.log.gz 2024-02-10T05:41:41.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49702.log 2024-02-10T05:41:41.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46980.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46980.log.gz 2024-02-10T05:41:41.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55118.log 2024-02-10T05:41:41.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.1.49702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55753.log 2024-02-10T05:41:41.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55118.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55118.log.gz 2024-02-10T05:41:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44051.log 2024-02-10T05:41:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55753.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55753.log.gz 2024-02-10T05:41:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40512.log 2024-02-10T05:41:41.432 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49310.log 2024-02-10T05:41:41.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40512.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40512.log.gz 2024-02-10T05:41:41.440 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54175.log 2024-02-10T05:41:41.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57079.log 2024-02-10T05:41:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54175.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54175.log.gz 2024-02-10T05:41:41.464 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53829.log 2024-02-10T05:41:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57079.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57079.log.gz 2024-02-10T05:41:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50831.log 2024-02-10T05:41:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53829.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53829.log.gz 2024-02-10T05:41:41.493 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48020.log 2024-02-10T05:41:41.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50831.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50831.log.gz 2024-02-10T05:41:41.502 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40710.log 2024-02-10T05:41:41.502 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48020.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48020.log.gz 2024-02-10T05:41:41.512 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41251.log.gz 2024-02-10T05:41:41.519 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49813.log.gz 2024-02-10T05:41:41.520 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47648.log 2024-02-10T05:41:41.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40710.log: 86.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40710.log.gz 2024-02-10T05:41:41.533 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51651.log 2024-02-10T05:41:41.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47648.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47648.log.gz 2024-02-10T05:41:41.542 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49979.log.gz 2024-02-10T05:41:41.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61447.log 2024-02-10T05:41:41.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51651.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51651.log.gz 2024-02-10T05:41:41.545 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57275.log 2024-02-10T05:41:41.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61447.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61447.log.gz 2024-02-10T05:41:41.545 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45548.log 2024-02-10T05:41:41.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57275.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57275.log.gz 2024-02-10T05:41:41.546 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60912.log 2024-02-10T05:41:41.550 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45548.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45548.log.gz 2024-02-10T05:41:41.552 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64584.log 2024-02-10T05:41:41.557 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60912.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51127.log.gz 2024-02-10T05:41:41.563 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60912.log.gz 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63987.log 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50274.log 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.64584.log.gz 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57299.log 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63987.log.gz 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50274.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50274.log.gz 2024-02-10T05:41:41.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60428.log 2024-02-10T05:41:41.569 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47670.log 2024-02-10T05:41:41.569 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57299.log.gz/var/log/ceph/cluster2-client.mirror.60428.log: 2024-02-10T05:41:41.569 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54327.log 2024-02-10T05:41:41.569 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47670.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60428.log.gz 2024-02-10T05:41:41.569 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48488.log 2024-02-10T05:41:41.570 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54327.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54327.log.gz 2024-02-10T05:41:41.571 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41378.log 2024-02-10T05:41:41.571 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48488.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48488.log.gz 2024-02-10T05:41:41.571 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47549.log 2024-02-10T05:41:41.571 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41378.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41378.log.gz 2024-02-10T05:41:41.571 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54375.log 2024-02-10T05:41:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47549.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47549.log.gz 2024-02-10T05:41:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57824.log 2024-02-10T05:41:41.573 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54375.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54375.log.gz 2024-02-10T05:41:41.573 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57362.log 2024-02-10T05:41:41.574 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57824.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57824.log.gz 2024-02-10T05:41:41.574 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48743.log 2024-02-10T05:41:41.575 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57362.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57362.log.gz 2024-02-10T05:41:41.575 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45184.log 2024-02-10T05:41:41.576 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48743.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48743.log.gz 2024-02-10T05:41:41.576 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47795.log 2024-02-10T05:41:41.577 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.45184.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.45184.log.gz 2024-02-10T05:41:41.577 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55825.log 2024-02-10T05:41:41.578 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47795.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47795.log.gz 2024-02-10T05:41:41.579 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47736.log 2024-02-10T05:41:41.580 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55825.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55825.log.gz 2024-02-10T05:41:41.580 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.38880.log 2024-02-10T05:41:41.580 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47736.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47736.log.gz 2024-02-10T05:41:41.581 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41229.log 2024-02-10T05:41:41.588 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.2.38880.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.38880.log.gz 2024-02-10T05:41:41.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53942.log 2024-02-10T05:41:41.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41229.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41229.log.gz 2024-02-10T05:41:41.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58567.log 2024-02-10T05:41:41.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53942.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53942.log.gz 2024-02-10T05:41:41.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50689.log 2024-02-10T05:41:41.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58567.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58567.log.gz 2024-02-10T05:41:41.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54416.log 2024-02-10T05:41:41.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50689.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50689.log.gz 2024-02-10T05:41:41.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52987.log 2024-02-10T05:41:41.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54416.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54416.log.gz 2024-02-10T05:41:41.598 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61867.log 2024-02-10T05:41:41.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52987.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52987.log.gz 2024-02-10T05:41:41.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43287.log 2024-02-10T05:41:41.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61867.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61867.log.gz 2024-02-10T05:41:41.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62727.log 2024-02-10T05:41:41.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43287.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43287.log.gz 2024-02-10T05:41:41.602 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42295.log 2024-02-10T05:41:41.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62727.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62727.log.gz 2024-02-10T05:41:41.603 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42359.log 2024-02-10T05:41:41.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42295.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42295.log.gz 2024-02-10T05:41:41.604 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56789.log 2024-02-10T05:41:41.609 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42359.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42359.log.gz 2024-02-10T05:41:41.610 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45992.log 2024-02-10T05:41:41.610 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56789.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56789.log.gz 2024-02-10T05:41:41.611 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37546.log 2024-02-10T05:41:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45992.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45992.log.gz 2024-02-10T05:41:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58305.log 2024-02-10T05:41:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37546.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37546.log.gz 2024-02-10T05:41:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63607.log 2024-02-10T05:41:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58305.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58305.log.gz 2024-02-10T05:41:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63787.log 2024-02-10T05:41:41.619 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63607.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63607.log.gz 2024-02-10T05:41:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45688.log 2024-02-10T05:41:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63787.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63787.log.gz 2024-02-10T05:41:41.621 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58872.log 2024-02-10T05:41:41.621 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45688.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45688.log.gz 2024-02-10T05:41:41.622 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56118.log 2024-02-10T05:41:41.622 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.58872.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58872.log.gz 2024-02-10T05:41:41.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57444.log 2024-02-10T05:41:41.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56118.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56118.log.gz 2024-02-10T05:41:41.624 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47445.log 2024-02-10T05:41:41.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57444.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57444.log.gz 2024-02-10T05:41:41.625 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39464.log 2024-02-10T05:41:41.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47445.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47445.log.gz 2024-02-10T05:41:41.627 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53964.log 2024-02-10T05:41:41.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39464.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.39464.log.gz 2024-02-10T05:41:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55801.log 2024-02-10T05:41:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53964.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53964.log.gz 2024-02-10T05:41:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50811.log 2024-02-10T05:41:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55801.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55801.log.gz 2024-02-10T05:41:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44815.log 2024-02-10T05:41:41.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38671.log 2024-02-10T05:41:41.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44815.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44815.log.gz 2024-02-10T05:41:41.656 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42719.log 2024-02-10T05:41:41.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.1.38671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44447.log 2024-02-10T05:41:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.42719.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42719.log.gz 2024-02-10T05:41:41.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44791.log 2024-02-10T05:41:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64209.log 2024-02-10T05:41:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44791.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44791.log.gz 2024-02-10T05:41:41.713 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62587.log 2024-02-10T05:41:41.713 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64209.log.gz 2024-02-10T05:41:41.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41937.log 2024-02-10T05:41:41.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62587.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62587.log.gz 2024-02-10T05:41:41.758 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50459.log 2024-02-10T05:41:41.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41937.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41937.log.gz 2024-02-10T05:41:41.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60360.log 2024-02-10T05:41:41.776 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50459.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50459.log.gz 2024-02-10T05:41:41.784 INFO:teuthology.orchestra.run.smithi060.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38671.log.gz 2024-02-10T05:41:41.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.38916.log 2024-02-10T05:41:41.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60360.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60360.log.gz 2024-02-10T05:41:41.801 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55251.log 2024-02-10T05:41:41.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.3.38916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55849.log 2024-02-10T05:41:41.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55251.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55251.log.gz 2024-02-10T05:41:41.817 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44051.log.gz 2024-02-10T05:41:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51755.log 2024-02-10T05:41:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55849.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55849.log.gz 2024-02-10T05:41:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60066.log 2024-02-10T05:41:41.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63107.log 2024-02-10T05:41:41.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60066.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60066.log.gz 2024-02-10T05:41:41.862 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64891.log 2024-02-10T05:41:41.862 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63107.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63107.log.gz 2024-02-10T05:41:41.874 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49389.log 2024-02-10T05:41:41.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64891.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64891.log.gz 2024-02-10T05:41:41.882 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47670.log.gz 2024-02-10T05:41:41.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55620.log 2024-02-10T05:41:41.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62527.log 2024-02-10T05:41:41.904 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55620.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55620.log.gz 2024-02-10T05:41:41.904 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41123.log 2024-02-10T05:41:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62527.log: 93.7% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62527.log.gz 2024-02-10T05:41:41.908 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49310.log.gz 2024-02-10T05:41:41.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54496.log 2024-02-10T05:41:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41123.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41123.log.gz 2024-02-10T05:41:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55683.log 2024-02-10T05:41:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54496.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54496.log.gz 2024-02-10T05:41:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63127.log 2024-02-10T05:41:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55683.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55683.log.gz 2024-02-10T05:41:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56721.log 2024-02-10T05:41:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63127.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63127.log.gz 2024-02-10T05:41:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60987.log 2024-02-10T05:41:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.56721.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56721.log.gz 2024-02-10T05:41:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44767.log 2024-02-10T05:41:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60987.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60987.log.gz 2024-02-10T05:41:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54034.log 2024-02-10T05:41:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44767.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44767.log.gz 2024-02-10T05:41:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51473.log 2024-02-10T05:41:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54034.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54034.log.gz 2024-02-10T05:41:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48616.log 2024-02-10T05:41:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51473.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51473.log.gz 2024-02-10T05:41:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47837.log 2024-02-10T05:41:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48616.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48616.log.gz 2024-02-10T05:41:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39447.log 2024-02-10T05:41:41.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47837.log: 94.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.38916.log.gz 2024-02-10T05:41:41.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39177.log 2024-02-10T05:41:41.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39447.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.39447.log.gz 2024-02-10T05:41:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61247.log 2024-02-10T05:41:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39177.log.gz 2024-02-10T05:41:41.950 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49268.log 2024-02-10T05:41:41.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61247.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61247.log.gz 2024-02-10T05:41:41.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64831.log 2024-02-10T05:41:41.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49268.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49268.log.gz 2024-02-10T05:41:41.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63347.log 2024-02-10T05:41:41.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64831.log: 9.2% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50811.log.gz 2024-02-10T05:41:41.981 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49409.log 2024-02-10T05:41:41.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63347.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63347.log.gz 2024-02-10T05:41:41.983 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64831.log.gz 2024-02-10T05:41:41.999 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.53206.log 2024-02-10T05:41:41.999 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49409.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49409.log.gz 2024-02-10T05:41:42.008 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49628.log.gz 2024-02-10T05:41:42.009 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44383.log 2024-02-10T05:41:42.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.53206.log: 95.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49702.log.gz 2024-02-10T05:41:42.034 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-02-10T05:41:42.035 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.44383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58392.log 2024-02-10T05:41:42.040 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-osd.0.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44383.log.gz 2024-02-10T05:41:42.054 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.53206.log.gz 2024-02-10T05:41:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47504.log 2024-02-10T05:41:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48901.log 2024-02-10T05:41:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47504.log: /var/log/ceph/cluster2-client.mirror.58392.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58392.log.gz 2024-02-10T05:41:42.065 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39310.log 2024-02-10T05:41:42.065 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48901.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48901.log.gz 2024-02-10T05:41:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57179.log 2024-02-10T05:41:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.39310.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39310.log.gz 2024-02-10T05:41:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39600.log 2024-02-10T05:41:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57179.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57179.log.gz 2024-02-10T05:41:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58767.log 2024-02-10T05:41:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.39600.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39600.log.gz 2024-02-10T05:41:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45734.log 2024-02-10T05:41:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58767.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58767.log.gz 2024-02-10T05:41:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38428.log 2024-02-10T05:41:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.45734.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62607.log 2024-02-10T05:41:42.140 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.38428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49833.log 2024-02-10T05:41:42.140 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38428.log.gz 2024-02-10T05:41:42.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62607.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45734.log.gz 2024-02-10T05:41:42.141 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62607.log.gz 2024-02-10T05:41:42.151 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38839.log 2024-02-10T05:41:42.151 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.49833.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49833.log.gz 2024-02-10T05:41:42.159 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54729.log 2024-02-10T05:41:42.169 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58988.log 2024-02-10T05:41:42.172 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.54729.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38839.log.gz 2024-02-10T05:41:42.177 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64911.log 2024-02-10T05:41:42.182 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58988.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54729.log.gz 2024-02-10T05:41:42.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61707.log 2024-02-10T05:41:42.191 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.64911.log: 9.2% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58988.log.gz 2024-02-10T05:41:42.195 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48348.log 2024-02-10T05:41:42.199 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61707.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.64911.log.gz 2024-02-10T05:41:42.205 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40823.log 2024-02-10T05:41:42.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48348.log: 79.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61707.log.gz 2024-02-10T05:41:42.217 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48348.log.gz 2024-02-10T05:41:42.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50989.log 2024-02-10T05:41:42.232 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52862.log 2024-02-10T05:41:42.245 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50989.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50989.log.gz 2024-02-10T05:41:42.245 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47904.log 2024-02-10T05:41:42.246 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52862.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52862.log.gz 2024-02-10T05:41:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59233.log 2024-02-10T05:41:42.256 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.47904.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47904.log.gz 2024-02-10T05:41:42.273 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48451.log 2024-02-10T05:41:42.274 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59233.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59233.log.gz 2024-02-10T05:41:42.282 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43393.log 2024-02-10T05:41:42.295 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.48451.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48451.log.gz 2024-02-10T05:41:42.298 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62467.log 2024-02-10T05:41:42.299 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.43393.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43393.log.gz 2024-02-10T05:41:42.309 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63367.log 2024-02-10T05:41:42.310 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62467.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62467.log.gz 2024-02-10T05:41:42.312 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51755.log.gz 2024-02-10T05:41:42.313 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39242.log 2024-02-10T05:41:42.314 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63367.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63367.log.gz 2024-02-10T05:41:42.324 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42901.log 2024-02-10T05:41:42.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39242.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.39242.log.gz 2024-02-10T05:41:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60215.log 2024-02-10T05:41:42.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42901.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62227.log 2024-02-10T05:41:42.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.60215.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60215.log.gz 2024-02-10T05:41:42.351 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42901.log.gz 2024-02-10T05:41:42.359 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61587.log 2024-02-10T05:41:42.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.62227.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62227.log.gz 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57761.log 2024-02-10T05:41:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61587.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61587.log.gz 2024-02-10T05:41:42.384 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57159.log 2024-02-10T05:41:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57761.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57761.log.gz 2024-02-10T05:41:42.389 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47504.log.gz 2024-02-10T05:41:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46508.log 2024-02-10T05:41:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.57159.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57159.log.gz 2024-02-10T05:41:42.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54058.log 2024-02-10T05:41:42.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46508.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46508.log.gz 2024-02-10T05:41:42.398 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64802.log 2024-02-10T05:41:42.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54058.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54058.log.gz 2024-02-10T05:41:42.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59545.log 2024-02-10T05:41:42.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64802.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64802.log.gz 2024-02-10T05:41:42.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50158.log 2024-02-10T05:41:42.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.59545.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59545.log.gz 2024-02-10T05:41:42.402 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54616.log 2024-02-10T05:41:42.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.50158.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50158.log.gz 2024-02-10T05:41:42.404 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55894.log 2024-02-10T05:41:42.404 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54616.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54616.log.gz 2024-02-10T05:41:42.404 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46880.log 2024-02-10T05:41:42.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.55894.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55894.log.gz 2024-02-10T05:41:42.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45464.log 2024-02-10T05:41:42.408 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.46880.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46880.log.gz 2024-02-10T05:41:42.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37322.log 2024-02-10T05:41:42.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45464.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45464.log.gz 2024-02-10T05:41:42.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.37322.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37322.log.gz 2024-02-10T05:41:42.414 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47837.log.gz 2024-02-10T05:41:42.427 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44447.log.gz 2024-02-10T05:41:42.455 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49389.log.gz 2024-02-10T05:41:42.743 INFO:teuthology.orchestra.run.smithi060.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40823.log.gz 2024-02-10T05:41:43.363 INFO:teuthology.orchestra.run.smithi008.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-02-10T05:41:43.905 INFO:teuthology.orchestra.run.smithi008.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-02-10T05:41:44.179 INFO:teuthology.orchestra.run.smithi008.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-02-10T05:41:44.181 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T05:41:44.181 INFO:teuthology.orchestra.run.smithi008.stderr:real 0m6.551s 2024-02-10T05:41:44.182 INFO:teuthology.orchestra.run.smithi008.stderr:user 0m19.107s 2024-02-10T05:41:44.182 INFO:teuthology.orchestra.run.smithi008.stderr:sys 0m0.668s 2024-02-10T05:42:31.845 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-02-10T05:42:32.508 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-02-10T05:42:47.422 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-02-10T05:42:47.424 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-02-10T05:42:47.424 INFO:teuthology.orchestra.run.smithi060.stderr:real 1m9.766s 2024-02-10T05:42:47.424 INFO:teuthology.orchestra.run.smithi060.stderr:user 3m14.485s 2024-02-10T05:42:47.425 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m7.222s 2024-02-10T05:42:47.425 INFO:tasks.ceph:Archiving logs... 2024-02-10T05:42:47.426 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi008/log 2024-02-10T05:42:47.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:42:50.151 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi060/log 2024-02-10T05:42:50.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:43:18.673 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi008 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2024-02-10T05:43:18.680 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T05:43:18.694 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-02-10T05:43:18.702 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T05:43:18.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T05:43:18.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T05:43:18.797 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T05:43:18.810 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T05:43:18.810 DEBUG:teuthology.orchestra.run.smithi008:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:43:18.813 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:43:18.826 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:43:18.829 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:43:18.832 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:43:18.832 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-02-10T05:43:18.832 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +618us[ +620us] +/- 88ms 2024-02-10T05:43:18.833 INFO:teuthology.orchestra.run.smithi008.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -728us[ -727us] +/- 62ms 2024-02-10T05:43:18.833 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 117 +467us[ +470us] +/- 78ms 2024-02-10T05:43:18.833 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:43:18.833 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:43:18.834 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-02-10T05:43:18.834 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +559us[ +559us] +/- 88ms 2024-02-10T05:43:18.834 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -725us[ -724us] +/- 62ms 2024-02-10T05:43:18.834 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 109 +453us[ +456us] +/- 78ms 2024-02-10T05:43:18.834 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:43:18.835 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T05:43:18.847 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T05:43:18.848 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T05:43:18.879 DEBUG:teuthology.orchestra.run.smithi008:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:43:18.951 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:43:19.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:43:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:43:19.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:43:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:43:19.089 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T05:43:19.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T05:43:19.120 INFO:teuthology.task.internal:Duration was 17382.490612 seconds 2024-02-10T05:43:19.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T05:43:19.154 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T05:43:19.154 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:43:19.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:43:19.200 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:43:19.201 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:43:19.548 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T05:43:19.548 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T05:43:19.549 DEBUG:teuthology.orchestra.run.smithi008:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:43:19.570 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-02-10T05:43:19.571 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:43:19.590 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T05:43:19.591 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:43:19.613 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:43:19.660 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T05:43:19.660 DEBUG:teuthology.orchestra.run.smithi008:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:43:19.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:43:19.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T05:43:19.831 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T05:43:19.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:43:19.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:43:19.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T05:43:19.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:43:19.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:43:19.923 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = core 2024-02-10T05:43:19.950 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-02-10T05:43:19.981 DEBUG:teuthology.orchestra.run.smithi008:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:43:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:43:19.998 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:43:20.035 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi060.front.sepia.ceph.com, flagging run as failed 2024-02-10T05:43:20.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T05:43:20.049 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T05:43:20.050 DEBUG:teuthology.misc:Transferring archived files from smithi008:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi008 2024-02-10T05:43:20.051 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:43:20.085 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi060 2024-02-10T05:43:20.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:44:05.239 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-02-10T05:44:05.334 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696/remote/smithi060/coredump/1707529146.59977.core.gz: gzip compressed data, was "1707529146.59977.core", last modified: Sat Feb 10 05:43:39 2024, max compression, original size modulo 2^32 736460800 2024-02-10T05:44:05.335 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-02-10T05:44:08.531 INFO:teuthology.task.internal: core looks like: /tmp/tmpsdhgucd0: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster1 --id mirror.0', real uid: 1000, effective uid: 1000, real gid: 1301, effective gid: 1301, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2024-02-10T05:44:08.606 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-02-10T05:44:08.607 DEBUG:teuthology.orchestra.run.smithi060:> which 'rbd-mirror --foreground --cluster cluster1 --id mirror.0' 2024-02-10T05:44:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/which: unrecognized option '--foreground' 2024-02-10T05:44:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/which: unrecognized option '--cluster' 2024-02-10T05:44:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/which: unrecognized option '--id' 2024-02-10T05:44:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/which: no cluster1 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-02-10T05:44:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:/usr/bin/which: no mirror.0 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-02-10T05:44:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-10T05:44:08.627 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/rbd-mirror 2024-02-10T05:44:08.628 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 2: "which 'rbd-mirror --foreground --cluster cluster1 --id mirror.0'" 2024-02-10T05:44:08.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T05:44:08.661 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T05:44:08.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T05:44:08.683 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T05:44:08.683 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:44:08.690 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:44:08.705 INFO:teuthology.orchestra.run.smithi008.stdout: 265293 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest 2024-02-10T05:44:08.705 INFO:teuthology.orchestra.run.smithi008.stdout: 265294 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest/archive 2024-02-10T05:44:08.705 INFO:teuthology.orchestra.run.smithi008.stdout: 265298 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:44:08.705 INFO:teuthology.orchestra.run.smithi008.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 10 00:53 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-10T05:44:08.706 INFO:teuthology.orchestra.run.smithi008.stdout: 265300 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 10 00:53 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-10T05:44:08.706 INFO:teuthology.orchestra.run.smithi008.stdout: 265299 184 -rw-r--r-- 1 ubuntu ubuntu 184407 Feb 10 05:43 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:44:08.706 INFO:teuthology.orchestra.run.smithi008.stdout: 403055 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest/archive/audit 2024-02-10T05:44:08.706 INFO:teuthology.orchestra.run.smithi008.stdout: 403337 120 -rw------- 1 ubuntu ubuntu 120256 Feb 10 05:43 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-10T05:44:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:44:08.707 INFO:teuthology.orchestra.run.smithi008.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-10T05:44:08.707 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi008 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-10T05:44:08.718 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T05:44:08.733 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T05:44:08.761 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2024-02-10T05:44:08.761 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T05:44:08.769 INFO:teuthology.orchestra.run.smithi060.stdout: 262166 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest 2024-02-10T05:44:08.769 INFO:teuthology.orchestra.run.smithi060.stdout: 403057 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 10 01:02 /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-02-10T05:44:08.769 INFO:teuthology.orchestra.run.smithi060.stdout: 403058 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 10 01:03 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-02-10T05:44:08.769 INFO:teuthology.orchestra.run.smithi060.stdout: 417142 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 01:03 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 262167 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest/archive 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 10 00:53 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 265294 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Feb 10 00:53 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 265293 184 -rw-r--r-- 1 ubuntu ubuntu 187288 Feb 10 05:43 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 403055 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:43 /home/ubuntu/cephtest/archive/audit 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 403059 124 -rw------- 1 ubuntu ubuntu 126339 Feb 10 05:43 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-10T05:44:08.770 INFO:teuthology.orchestra.run.smithi060.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 01:39 /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:44:08.771 INFO:teuthology.orchestra.run.smithi060.stdout: 268198 47684 -rw------- 1 ubuntu ubuntu 639733760 Feb 10 01:38 /home/ubuntu/cephtest/archive/coredump/1707529131.57951.core 2024-02-10T05:44:08.771 INFO:teuthology.orchestra.run.smithi060.stdout: 268199 62288 -rw------- 1 ubuntu ubuntu 736460800 Feb 10 01:39 /home/ubuntu/cephtest/archive/coredump/1707529146.59977.core 2024-02-10T05:44:08.772 INFO:teuthology.orchestra.run.smithi060.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-10T05:44:08.789 DEBUG:teuthology.nuke:shortname: smithi008 2024-02-10T05:44:08.790 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:44:08.807 DEBUG:teuthology.nuke:shortname: smithi060 2024-02-10T05:44:08.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:44:08.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 00:45:09.209043', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlSAKYrhH4t6lqydJwPc7dakwrUcEIWsSCqIlLXUX8KcGMzn9wmRKv3395yFBqkRi00GYbssmuqVssZshiM65E='} 2024-02-10T05:44:08.826 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:39:00-rbd-main-distro-default-smithi/7554696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 00:45:09.207217', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3fOJ0AL93zBAlc9KUWLcYTG+/ZN22p+OeOUdu5BJmCIqUtDwSyLdMZQynzTw6sRiugO8+IUpLWGgYqOxmkkzs='} 2024-02-10T05:44:08.853 INFO:teuthology.orchestra.console.smithi008:Power off 2024-02-10T05:44:08.853 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:44:08.870 INFO:teuthology.orchestra.console.smithi060:Power off 2024-02-10T05:44:08.870 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:44:08.880 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2024-02-10T05:44:08.880 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:44:08.894 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-02-10T05:44:08.894 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:44:08.910 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-02-10T05:44:08.918 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-02-10T05:44:12.911 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:44:12.920 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:44:13.023 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-02-10T05:44:13.029 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-02-10T05:44:17.028 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:44:17.037 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:44:17.140 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-02-10T05:44:17.147 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-02-10T05:44:17.241 INFO:teuthology.orchestra.console.smithi008:Power off completed 2024-02-10T05:44:17.248 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-02-10T05:44:17.355 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 17382.49061179161 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi060 with status 110: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=76ee3d9f257446519d683777ce9bf050 status: fail success: false 2024-02-10T05:44:17.355 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:44:17.435 INFO:teuthology.run:FAIL